Frequency of Student Resource Use and Academic Performance in Preclerkship Education: A Survey Study

Medical students have unprecedented access to a large variety of learning resources, but patterns of resource use, differences in use across education cohorts, and the relationship between resource use and academic performance are unclear. Therefore, the purpose of the current study was to evaluate student resource use and its relationship to academic performance during preclerkship years. First-year and second-year medical students completed a 10-question electronic survey that assessed likelihood of using outside resources recommended by others, reasons for using outside resources, frequency of use of resources, and use of outside resources for specific disciplines. Outcomes were compared between the 2 cohorts of students. First-year students were more likely to use instructor-produced resources and self-generated study resources, and second-year students were more likely to use board review resources. Although differences were found between cohorts for frequency of use of certain resources, correlations between resource use and academic performance were modest. Overall, our results indicated that student use of study resources changed between the first and second years of medical school. These results suggest opportunities for medical educators to guide students in the selection and effective use of outside resources as they mature as self-regulated learners. Further, since students seem to extensively use external resources for learning, institutions should consider calibrating their curriculum and teaching methods to this learning style and providing high-quality, accessible resource materials for all students to reduce the potential impact of socioeconomic factors on student performance.


Introduction
The landscape of medical education is shifting, and students have an unprecedented number of resources and information available to them. As such, many students are abandoning traditional didactic methods of in-person classroom learning, and the exigencies of the COVID-19 pandemic have accelerated these changes [1,2]. Concurrently, the field of medical education is embracing the concept of self-regulated learning, whereby students develop personalized strategies for independent learning, such as selection of resources and use feedback to achieve desired academic outcomes [3][4][5]. Therefore, students need to learn how to select, critically evaluate, and effectively use information and resources from sources outside the classroom.
In addition to using resources provided by instructors in their academic courses, medical students typically include other resources as part of their study strategy. For example, they may use outside resources from the Internet, board preparation resources, and notes and charts from other students. Shershneva et al. [6] noted that effective use of resources was an important skill for medical training and practice. Given that students are attending in-person classes less frequently and considering the theoretical framework of self-regulated learning, it is important to understand how students use resources to inform us how they apply various learning strategies to achieve academic goals [1].
Research investigating resource use by medical students found a range of factors that influenced resource selection, such as ease of access and perceived quality of the information [7][8][9]. In a study of a single course [10], medical and dental student resource use did not vary significantly by learning resource type. In another study [11], the use of review books and Internet resources during a clerkship was significantly higher than the use of the recommended textbook. Other studies have reported positive correlations between resource use and exam performance [12,13].
At the institution of the current study, osteopathic medical students progress through an integrated systems-based curriculum and distributed model, learning in community health centers during their second year. The curriculum of the first year is primarily lecture based (in-person and/or recorded) with active learning sessions. The students are primarily assessed with licensing exam style-like questions developed by the faculty. In courses, some instructors provided required or recommended readings and this is determined by the individual instructor. Although faculty have some awareness of student resource use through anecdotal interactions and feedback from advisees, patterns of resource use during preclerkship years, such as frequency of use of various resource types and reasons for use of outside resources, are unknown. Further, the relationship between resource use and academic performance is unclear. Therefore, the purpose of the current study was to evaluate student resource use and its relationship to academic performance during preclerkship years. A better understanding of student resource use and selection may be informative for instructors and result in improved curricular development and student advising.

Study Design and Participants
The current survey study was designed to determine the frequency of use of various study resources by first-year and second-year medical students and whether resource use was associated with academic performance. The study was conducted during the 2019-2020 academic year. An e-mail invitation with a link to an electronic survey was sent to first-year students in October 2019 and second-year students in January 2020. The survey instructions informed students about the study purpose and indicated that participation was voluntary with no direct benefit and that they could stop participating at any time. They were also informed that they would be required to provide identifying information in the survey so responses could be linked with academic records. Study investigators were blinded to identifying information, and student confidentiality was preserved during analysis and data reporting. The survey took about 15 min to complete. Students had 2 months to complete the survey and were sent 2 reminder e-mails. The A.T. Still University-Arizona Institutional Review Board approved this study #2019-116. Students provided their consent by completing the survey.

Survey Design
An electronic survey (Qualtrics, Seattle, WA) with 10 questions was created specifically for the current study (Appendix 1). A group composed of faculty and students reviewed the survey for face and content validity. Four questions collected demographic information about name, age, sex, and highest degree attained before medical school matriculation. Four questions used Likert-type responses to assess likelihood of using outside resources (not recommended for class) recommended by others (e.g., instructors, classmates, and Internet), reasons for using outside resources (e.g., exam preparation, more detail, and practice questions), frequency of use of resources (instructor-produced resources, outside resources, student-generated resources), and use of outside resources for specific disciplines. Two open-ended questions asked about other reasons for using outside resources and for additional comments about the use of outside resources.

Statistical Analysis
Survey responses were summarized using frequency and percentage. Mann-Whitney tests were used to compare Likert-type responses between first-year and second-year students. Spearman correlation coefficients were used to estimate the strength of associations between resource use and academic performance. For academic performance, data were collected from students' academic records for course grades and from Comprehensive Osteopathic Medical Licensing Examination of the United States (COMLEX-USA) Level 1 scores for second-year students. Student grade point average (GPA) was used for analyses and was obtained and coded by a data informatics specialist with Family Educational Rights and Privacy Act authorization. The GPA data were considered separately and collectively for both years for analyses. Because of the large number of comparisons, a familywise error approach [14] was used to control for potential inflation of type I errors. Specifically, a Bonferroni correction was used, so the α level did not exceed .05 divided by the number of tests in each comparison. For all other analyses, an α of .05 (2-tailed) was used for statistical significance. SPSS version 26 (IBM Corp., Armonk, NY) was used for all data analyses.

Use of Outside Resources Recommended by Others
For survey items about outside resources recommended by others, responses of very likely and extremely likely were grouped as likely for analysis. The majority of first-year students indicated they were likely to use outside resources recommended by instructors (33/78, 42.3%), classmates (51/79, 64.5%), and the Internet (42/79, 53.1%) ( Table 2). Conversely, a minority of second-year students indicated they were likely to use outside resources recommended by instructors (13/58, 22.4%), but a majority were likely to use those recommended by classmates (42/58, 72.4%) and the Internet (28/58, 48.3%). Data for use of outside resources for specific disciplines are presented in Appendix 2.

Reasons for Using Outside Resources
Reasons for using outside resources are presented in Table 3; responses of somewhat agree and strongly agree were grouped for analysis. Most students indicated that they used Differences were found between cohorts for these 2 reasons (P = .002 and P = .001, respectively) ( Fig. 1). No other differences were found between cohorts (all P > .05).

Frequency of Use of Resources
For survey items about frequency of use, responses of daily and several times a week were grouped as frequently for analysis. Data for use of instructor-produced resources are presented in Table 4. First-year students more frequently used instructor resources than second-year students. These included instructor slides, transcripts, recorded presentations, practice questions from instructors, and other instructor-created resources (P < .001, P = .002, P < .001, P < .001, and P < .001, respectively) ( Fig. 2 Fig. 3). When considering both cohorts collectively, weak negative correlations were found between GPA and search engine (r s = −.20,  Table 5). When considering both cohorts collectively, modest positive correlations were found between GPA and use of board review resources (r s = .31, P = .001) and board review question banks (r s = .38, P < .001). Among first-year students, a modest negative correlation was found between GPA and use of board review question banks (r s = −.25, P = .047).  Regarding use of student-generated resources, first-year students used self-generated study resources, such as flashcards or charts, more frequently than second-year students (75.3% [55/73] vs 51.9% [28/54], P < .01) ( Table 6 and Fig. 5). When comparing frequency of the different resources used and COMLEX-USA Level 1 performance for secondyear students, only 1 negatively correlated predictor was found between student use of outside student-generated resources and COMLEX-USA Level 1 performance (r s = −.32, P = .04). No other differences were found (all P > .05).

Open-ended Survey Responses
Twenty-seven student comments, 14 OMS I and 13 OMS II, were captured by the 2 open-ended survey questions. In response to the question about other reasons for using outside resources, a few students indicated that they thought instructor-produced lectures were insufficient for their learning needs. One first-year student wrote that they used other resources because they were "looking for condensed versions of the material. Lectures can sometimes feel disorganized." A second-year student wrote, "In general, I found they are better at explaining important details. Some lecturers focus on the details too much and forget to teach the main concept." For the question asking for additional comments about use of outside resources, 2 students indicated that outside resources provided a way to supplement instructor-provided material. A first-year student wrote, "I prefer to get a very generalized view of a topic before watching lecture material. So oftentimes I will watch a youtube [sic] video prior to using course material." A second-year student wrote, "I use outside resources to get better understanding of things that I did not fully understand from the lectures." Other students considered the use of outside resources as the best way to prepare for board exams. One student wrote, "I enjoy our lectures, but sometimes feel that the high-yield board exam concepts are not always stressed in lecture. For that reason, I rely on outside resources to focus my studying to make sure I am fully understanding the material I will be heavily tested on come boards time!" Another wrote, "I cross-check everything against First Aid. If it's not in First Aid, I generally don't learn it." Conversely, some students indicated that they preferred interacting with the instructor and using instructor-produced resources. A first-year student wrote that they "don't always use outside resources because coursework contains unique details that don't comport with outside prep materials." A second-year student wrote, "I find that if I don't understand a concept, I would rather talk to the professor before looking at other outside resources."

Discussion
In the current study, we evaluated student resource use and its relationship to academic performance during preclerkship years. As expected, first-year and second-year medical students frequently used outside resources to study. Electronic resources, such as search engines and online videos, were used almost every day, and we found a modest negative correlation between search engine use and GPA. Further, second-year students were more likely to use outside board review resources than first-year students. Use of these resources showed modest positive correlations with GPA for both groups of students and weak negative correlations for first-year students only. First-year students were also more likely to use instructor-produced resources and selfgenerated study resources than second-year students. Overall, our results indicated that student use of study resources changes between the first and second years of medical school.   Our findings about frequency of use of outside resources, especially electronic resources, are consistent with previous studies [15,16]. However, the weak negative correlations between GPA and frequency of use of general search engines (e.g., Google) and Wikipedia in the current study suggested that these resources may not be effective study tools. A possible explanation for the poorer academic performance of those using these outside resources may be related to the quality of the information. According to Choi-Lundberg et al. [9], students who more frequently rely on search engines or Wikipedia may find less accurate information than in primary resources. In another study, Judd and Elliott [17] found that first-year students frequently used Google and Wikipedia to study even though they did not rate the quality of the information as highly as other resources. Perhaps, students are using these outside electronic resources because they believe that e-learning resources are more efficient for locating and learning highyield information [18]. Further, if instructor-produced resources are difficult to search, students may perceive that lack of efficiency as a barrier to their use. Therefore, faculty should consider the volume and organization of information when designing content for student use. This suggestion is supported by research that recommends faculty provide materials that maximize student learning [19]. Alternatively, experienced faculty and students could use their expertise to improve the content quality of existing external resources commonly used by students [20,21].
In the current study, our finding that first-year students used instructor-produced resources more frequently than second-year students was expected. As the first-year students adapted to the challenges of medical school, it seems likely that they preferred to become familiar with the instructorproduced and institution-provided resources before seeking outside resources. Specifically, our first-year students indicated that they were more likely to use recorded presentations, practice questions, handouts, and transcripts to study and that they were less likely to use assigned or recommended readings, instructor-led review sessions, and office hours or other instructor interactions. In general, review sessions and office hours occur less frequently, which may explain their lower frequency of use. Further, less frequent use of assigned or recommended readings is consistent with other studies [22,23]. Additional research suggests students are less motivated to use textbooks and instructor-suggested readings when electronic resources are available and are perceived as more efficient [15,20].
Our study findings may also represent a shift in focus as students progress through their medical education, where second-year students become less reliant on instructorproduced resources. For example, our second-year students reported more frequent use of board review resources and board review question banks than first-year students. This finding is expected given that second-year students are required to take board exams at the end of the academic year. It is also consistent with responses from second-year  Second-year students indicated they were more motivated to use outside resources to obtain additional detail, seek efficient and effective learning resources, and identify additional practice questions. Similar to our study, Scott et al. [15] also found students used digital resources for practice questions.
In the current study, we found that students used selfgenerated resources more frequently than student-generated resources. This finding probably reflects the practice of students focusing on the material they produce before using resources generated by their peers. We also found that self-generated and student-generated resources were used more frequently by first-year students than second-year students. Again, this finding likely reflects a shift in study focus as second-year students prepare for board exams and prioritize use of board review resources over individual or studentgenerated resources.
The more frequent use of outside resources by secondyear students as they prepared for board exams was expected in the current study. Further, the negative correlation between the GPA of first-year students and frequency of use of outside practice questions and board review question banks suggested that an early focus on board preparation  OMS I, first-year osteopathic medical student; OMS II, second-year osteopathic medical student may distract students from other aspects of medical education that support academic performance. For instance, the use of board resources to study during the preclerkship curriculum may interfere with acquisition of foundational concepts. Further, board resources are typically based on review and should not be used to acquire new knowledge, which may explain the poorer academic performance. Since board review resources are not designed to teach new material, perhaps first-year students should be discouraged from using them. Although students who reported a higher use of question banks with multiple-choice questions had higher exam performance including on licensing exams [16,24], the skills required for better exam performance may not be correlated with the skills needed to perform clinical responsibilities [13]. This finding suggests that the skills that enable an individual to perform well on a multiple-choice licensing examination may not correlate with the skills needed to perform as a competent physician.
The results of the current study may be especially important for educational environments where self-directed and independent learning are incorporated into the curriculum. Further, academic libraries may also find our results useful when considering new additions to the library collection. Since our study suggested that students frequently used electronic study resources, libraries should consider prioritizing additional online resources over print resources. For instance, review resources, exam preparation resources, or videos could be added to collections instead of new eBooks and journal subscriptions.
The results of the current study should be interpreted within the context of several limitations. Since it was conducted at a single institution, results may not be widely generalizable. Further, the current study only measured the frequency of resource use; thus, other considerations, such as how resources are used, were not measured. Another limitation may be related to self-selection bias. Because the survey was distributed electronically, students who were more likely to use electronic outside resources may have been more likely to complete our survey. In addition, this study relies on self-reported student data which may not match actual student resource use.
Our results highlight the need for faculty to intentionally design learning sessions and curricula that promote the maturity of their students as self-regulated learners. Because access to numerous high-quality external resources, such as question banks and review videos, may be cost prohibitive to students from lower socioeconomic households, our observed preference for use of outside resources to study may exacerbate inequalities that exist among medical students [25]. Therefore, it is incumbent on faculty and institutions to produce and provide high-quality resources that are accessible to all students. Further, faculty and institutions should guide students on how to select and evaluate study resources that allow them to proficiently use learning resources as they develop as self-regulated learners.

Conclusion
Results of the current study suggested that patterns of resource use by first-year and second-year medical students differed across the preclerkship years. Further, the frequency of resource use was not, in general, correlated with academic performance. Overall, our study may advance opportunities for medical educators to guide students in the selection and use outside resources as they mature as self-regulated learners. As external resources become more abundant and easily accessible to medical students, institutions should consider their role in the development of their students as self-regulated learners and self-directed clinicians.

Survey Participation Information
The attached survey is for research conducted by the School of Osteopathic Medicine in Arizona (SOMA) at A. T. Still University. The purpose of this study is to increase understanding of student resource use and its relationship to academic performance. Your participation is voluntary and will only involve completing the questionnaire. Completing this questionnaire should require 15 min of your time. Please do not use any outside assistance while answering these questions. Please answer each question honestly and to the best of your ability. You will be asked to identify yourself on the questionnaire, which will allow survey responses to be linked with your academic records. All information abstracted from your academic records including course grades and discipline scores as well as this survey, will be coded by a Family Educational Rights and Privacy Act (FERPA)-trained individual to remove names and other identifying information before transferring the data to the principal investigators. No names or other identifying information will be used in discussing or reporting data. The results of this research might be published. Any research reports or publications resulting from this research will not reveal your name or identity. All of your data will remain confidential. There are no direct benefits to you for completing this questionnaire. By completing the questionnaire, you agree to allow your survey responses and academic records to be used for this study. Your participation is voluntary, and you may refuse to complete the questionnaire or stop at any time if you wish. ________________________________________________________________________________ Study resources (ex. flashcards, charts, or notes) generated by other students Practice questions generated by other students Online content generated by other students (ex. Anki cards) Self-generated study resources (flashcards, charts, notes, etc.) Self-generated online study resources (ex. Anki cards) How often do you use outside resources for the following disciplines? (Answer in table below). Response numbers of survey questions varied because students did not answer all questions