Introduction

A relatively incipient endeavor in the domain of clinical and translational research, community stakeholder engagement is a multi-disciplinary approach where patients, caregivers, patient advocates or members of the general public collaborate with researchers to ensure that research efforts reflect the needs of the population targeted by the health research (Domecq et al., 2014; Stallings et al., 2019). While different models, frameworks and taxonomies have been proposed to conceptualize key definitions and approaches related to stakeholder engagement, some common principles include: a) authentic and sustained engagement across the research continuum and beyond, b) clarity in the roles and expectations of all parties engaged in the research, c) mutual trust and respect, d) commitment to co-learning and co-production and e) access to the appropriate resources, supports and training (Banner et al., 2019). In the present study, the Citizen Science (CS) program is conceived as an example of stakeholder engagement.

The CS program includes a diverse array of community members serving as Citizen Scientists (CSs) who participate in healthcare research projects to ensure the voice of the community is heard and understood by the healthcare researchers. To support CSs’ participation in a wide range of research projects, the design and development of specific curricula are necessary as CSs need a foundation in healthcare-related research topics and terminology. Our first CS curriculum addressed a broad range of topics to prepare the CSs to meaningfully contribute to healthcare research projects (Valle et al., 2018). Within the present work, we extended this CS curriculum program in response to changing research priorities of the institution to prepare CSs to collaborate to cancer-related research. Despite the focus on “access to the appropriate resources, supports and training” (Banner et al., 2019, p. 5), the operationalization of resources and training to support community stakeholder engagement initiatives is still under-represented in the literature, with only a few exceptions (McElfish et al., 2019; Valle et al., 2018). Given the lack of models and design strategies empirically tested and validated to address community stakeholder engagement initiatives, the present study provides some empirical basis to guide the facilitation of learning and performance in this particular context (Richey & Klein, 2014). More specifically, the present study describes the full analysis, design, development and formative evaluation of the Citizen Science Cancer Curriculum (CSCC). The curriculum is part of a community engagement initiative (Citizen Science Program) housed at a large public research university in the southeast United States.

Learner Analysis, Motivation and Instructional Goals

As any instructional design initiative, we first sought to better understand the needs of the target learners – CSs aspiring to contribute to cancer-related research – and the needs of the cancer researchers who would work with the CSs. We sought to intentionally match the needs of the learners and the cancer researchers to create a relevant learning experience to ensure the mutual goals addressed with the curriculum. Therefore, we conducted a needs assessment with the two parties to explore their mutual goals. Seven CSs and three researchers participated in different parts of the study. The needs assessment involved a focus group (duration of 49 min) with three learners and one researcher, individual interviews with three learners (M = 34.33 min) and three researchers (M = 41.66 min) and a short survey completed by all participants. We discovered learners’ initial motivation to engage in cancer-related research as well as their thoughts about the development of the curriculum and its learning materials.

Citizen Scientists

When we asked the CSs about their motivation for working on cancer-related research projects, we found that their motivation was related to: a) a sense of social responsibility, b) the value they attribute to their work as CSs and c) personal stories and patient status. For example, in terms of social responsibility, one participant said that her motivations to work on cancer-related projects is because “cancer affects so many people”:

This is probably a general motivation that you'll hear from anybody, but cancer affects so many people. And so, you know, I have people on my personal life that have cancer, and have been fighting it and it just seems like such a horrible, horrible situation and our only hope is research, you know, and so it's kind of like whatever you can do to jump on board and help with the hope that we have is, is what you do. Yeah, that's, that's my motivation.

Another participant mentioned her own health condition, patient status and other personal reasons for her motivation (e.g., grandfather who died from leukemia, a cousin who died from breast cancer). She described her motivation as being “a personal thing”:

I've lost my grandfather to leukemia many years ago, I had a cousin who died from breast cancer a few years ago. And so [...] just being on the personal side a little more. […] This is why we're going to do that. So now [it] is a personal thing for me.

Cancer Researchers

When we asked researchers to describe their motivation for working with CSs on cancer-related projects, they: a) mentioned how CSs can help bridge communication between scientists and the community, b) emphasized the need for accountability in face of existing disparities between communities and c) discussed the challenges of creating partnerships between researchers and communities to support robust cancer-related research. For instance, in relation to enhancing communication between the scientific community and community stakeholders, one of the researchers said:

Because I think a citizen scientist has the potential of being the connection with the community in the sense that we don't stay in those silos of the scientists over here or the physicians over here and then the community over there.

Another researcher described his views that, as a country, the U.S. innovates, but these innovations are often not accessible to the people who fund them or who are part of the community in which those discoveries are made.

We use this (cancer research) to better human conditions, but at the same time, there are some political things, some educational barriers in our society, which doesn't allow to apply this in our own population […] and our patients can simply not afford it. And I think the only way to change this on the long run is to really to have people, which are better than we as scientists and more centered in their communities. So to bring this up, this is nothing people talk about.

He mentioned the challenge of a scientist not being able to fully get the message in the hands of the community, which highlights why CS involvement is vital to mitigate communication barriers:

So that, that's why I think it's really [sic] to have that link and I think I want to help. I will […] if we find the right tool and I have confidence from people which work in the community that we're hitting the right tone, so that we make a difference […] if we can work together with a citizen scientist, then we can amplify these efforts because there's only so much you can do as a scientist.

Reconciling CSs’ Learning Needs and Researchers’ Instructional Goals

Reflecting on the original CS curriculum (Valle et al., 2018), most CSs mentioned video tutorials, case studies and storytelling as the appropriate instructional strategy for the curriculum. At least two participants also suggested the use of reflective questions to support learning and to provide emotional support:

[…] there's a lot of power in those interview videos and the all the Citizen Scientists in action videos (from previous curriculum) […] One of the main powers of those videos I think is that it helps you to process the information in that you can kind of take a step down from maybe an academic type of learning and kind of have more of a social type of learning if that makes sense. I think that that's something that will be really necessary with this topic […] the lectures and then maybe some videos to watch. And then maybe the quizzes could be a combination of like a quiz on the information and then a quiz of like, kind of like a check in on how you're feeling quiz, maybe questions like,’ how do you feel about this?’ […] What do you feel was your level of understanding of this topic that was just explained?’ I don't know, provide some sort of source for people to kind of check in with themselves. […] I just think it allows them to build their own checkpoints […] this is what I know, this is what I need to work on, this is what’s stressing out about the information, you know, this is what I feel confident about and I’d be willing to go talk to a researcher about. Maybe adding in some "emotional quizzes".

When asked about which topics they would like to see in the cancer curriculum, the CSs emphasized the importance of foundational concepts and terminology as well as more specific details about causes and treatments of cancer. The following list provides a summary of the more frequent topics mentioned by the CSs through the interviews and survey: introduction to cancer, basic concepts (cellular and genetic level), causes, diagnostic process, treatments, the cancer lexicon, types of cancers and survival rates, different populations and perspectives from cancer patients and caregivers.

When asked about which instructional goals are deemed necessary to prepare CSs for meaningful engagement on cancer-related research projects, the researchers mentioned some foundational knowledge and more detailed content based on CSs’ interest in and motivation to work on cancer-related projects as illustrated by the following feedback:

It's kind of like in like a pyramid. Okay, start broad, get a little bit more specific about what's actually going on in the lab and then even more detailed. So as far as what they need to know to be participants in this: they have to have an interest in science, I guess, but I don't think there's a huge prerequisite for this to be successful.[…] Now, the level of experience and the level of understanding of that topic, that will grow as… if they're interested in it, then I can give them more information, we can get more deeply involved in it.

Similar to the feedback provided by CSs, the researchers suggested a variety of topics to be integrated into the curriculum, including: basic concepts about cancer, viruses, risk factors (e.g., smoking), aspects related to scientific rigor, data quality control, reproducibility and scientific merit, clinical trial design, how new therapies are introduced into clinical practice, metastasis, genetics and cancer patients and caregivers. Given the outcomes of the needs assessment and the role of motivation in supporting collaboration intents between CSs and researchers, we elected to create a theory-based conceptual framework to guide the design and development efforts of the curriculum.

Conceptual Framework

The instructional approach described in this design and development case study was informed by the Self-Determination Theory (SDT). The study examined how the design, development and formative evaluation of an innovative curriculum could be used to support community stakeholder engagement in the context of cancer-related research. The SDT theory posits that motivation can be developed and sustained only when people’s innate need for autonomy, competence and relatedness are met (Ryan & Deci, 2017). SDT provides a useful framework for the design and development of the CSCC because it assumes an interconnection between people’s behavior and motivation (Valle et al., 2020). Furthermore, this framework recognizes the complexity of human motivation beyond dichotomic views. In other words, motivation is seen within a continuum that involves different levels of regulatory styles (external, introjected, identified and integrated) and loci of causality (impersonal, external and internal) as discussed by Ryan and Deci (2017). Finally, the motivation continuum assumed by the SDT framework is more likely to align with the range of motivations community members bring with them as they become part of the CS Program. Thus, the SDT framework was used to support CSs’ sense of autonomy, confidence and well-being – important psychological benefits. Figure 1 provides a visual of the conceptual framework.

Fig. 1
figure 1

Conceptual framework for CSCC using Self-Determination Theory

Full size image

Storytelling to Support Learning and Motivation

As a culturally universal product (Landrum et al., 2019), storytelling has been used as a pedagogical approach across different disciplines and contexts, including science, social science, psychology and patient self-management (Landrum et al., 2019; Lugmayr et al., 2017). Storytelling supports instructional strategies because stories can be used to: a) create interest, b) provide a structure for remembering course material, c) share information in a familiar and accessible form and d) create a more personal student-teacher connection (Green, 2004; Landrum et al., 2019). Furthermore, stories provide relatable examples and memory links that support recall, learning and problem-solving (Hung et al., 2012; Jonassen & Hernandez-Serrano, 2002).

In addition to its pedagogical value, storytelling can also support motivation by helping learners: a) become involved in the learning situation (Hung et al., 2012), b) make connections with their personal experiences (Yang & Wu, 2012), c) develop/sustain positive attitudes toward science (Valle et al., 2021) and d) improve perceptions of social representation (Valle et al., 2020), task value and self-efficacy (Yang & Wu, 2012). In the context of the curriculum, the motivational role of storytelling was expected to support learners’ motivation to act (e.g., become involved in cancer-related research) based on how they perceive their agency over their learning (autonomy), ability to contribute to cancer-related research (competence) and how they perceived themselves as part of the multi-disciplinary research group (relatedness).

Design of Citizen Science Cancer Curriculum

Our approach was largely informed by our conceptual framework, which addresses the motivational, emotional and cognitive outcomes of the curriculum. Our conception of the CSCC was beyond mere knowledge acquisition, as we intended to touch on the emotional and motivational states of the learners with authentic and relevant storytelling and an engaging emotional design. We outline these affordances within this section and discuss the development of these resources using emerging learning technologies. The full CSCC includes five modules that blends the educational affordances and key cancer-related topics thematically into the full, coherent, online, self-paced curriculum. The final CSCC is presented as an Open Educational Resource (OER) and distributed under a Creative Commons license with attribution, non-commercial use and share alike (Creative Commons, n.d.). The decision for creating the curriculum as an OER had important design implications, including the need for keeping the curriculum as free of branding as possible to facilitate its use by other institutions and prioritizing the utilization of more general topics rather than region-specific topics to ensure that the learning materials would be relevant to programs in other regions. Table 1 provides the module overview with the contents and activity types within each module. Figure 2 provides a visual of a typical lesson within a module of the program.

Table 1 Module overview with contents and activity types
Full size table
Fig. 2
figure 2

Example instructional video with thought questions, learning objectives, assessment and reflection questions

Full size image

Learning Objectives and Thought Questions

Each instructional video within a module includes clearly stated learning objectives and thought questions serving as pre-instructional strategies to heighten a learner’s awareness of the topics addressed within the lesson and to focus the learner’s attention on salient details and vocabulary (Hartley & Davies, 1976; Morrison et al., 2019). The writing of the learning objectives resulted from consultations with the subject-matter experts (SMEs) and followed best practices from instructional design literature. Each objective included actionable and measurable verbs connected to the subject matter under consideration. Following standard instructional design practices, the learning objectives were used to craft the embedded assessment items in an effort to clearly connect the objectives and assessment activities (Dick et al., 2015). In distinction, the thought questions were analogous to simultaneously provide an advanced organizer (Barnes & Clawson, 1975) for cuing the structure of the forthcoming materials and to provide relevance to the learners.

Instructional Videos and Supplemental Videos

We intentionally designed two types of videos for the CSCC: 1) instructional videos and 2) supplemental videos. All videos were created following best practices and evidence and are intentionally short (no more than 10 min) on well-defined topics with controls for stopping, playing, rewinding and fast-forwarding (Fiorella & Mayer, 2018; Mayer et al., 2020). The instructional videos are integrated with the learning objectives, thought questions, interactive embedded assessments and reflection questions; and serve as the primary intended learning outcomes connected to the topics identified in the needs assessment. Each of these instructional videos are presented by a SME and are recorded on a green screen with PowerPoint slides designed with the inspiration from the multimedia principles from the Cognitive Theory of Multimedia Learning (CTML) (Moreno & Mayer, 1999). The SMEs were encouraged to use a conversational style in the instructional videos and avoid the use of extraneous details within their explanations (coherence principle). The video scripts were created with the SMEs and an instructional designer to achieve these objectives and the slides made use of relevant images (multimedia principle) inclusive of both representational and organizational pictures (Levin, 1981) with minimal verbal redundancy (redundancy principle). Appropriate slides design involved placing imagery with verbal explanations concurrently (spatial and temporal contiguity principles) and included cues (e.g., bold or italics) (signaling principle) to emphasize key points in the narrative (modality principle). While the inclusion of the speaker on-screen in an instructional video remains an area of ongoing research (Wang & Antonenko, 2017), our intentions were to showcase diversity (e.g., gender, ethnicity and age) among the speakers to reach a broad audience of CSs. In contrast, the supplemental videos provided additional enrichment on topics that were not essential to the training of a CS to effectively participate in cancer-related research projects based on our needs assessment. The supplemental videos were designed with the same guiding principles CTML, but did not include the interactive embedded assessments or always make use of PowerPoint slides. The supplemental videos enrich the curriculum with additional relevant examples and discussion points and intentionally involved the use of storytelling.

To support CS’s motivation and self-efficacy to contribute to cancer-related research, the instructional design team met with presenters of both supplemental and instructional videos to discuss how the content in the videos could highlight concrete examples (storytelling) and the role CSs can play as part of a multi-disciplinary approach in cancer-related research. For example, in the supplemental video “Mapping Cancer Health Outcomes and Disparities”, the presenter introduced the topic by saying:

For cancer population sciences, maps are very important for describing and understanding the disparity in cancer health outcomes of our communities. As a Citizen Scientist, your research team is likely to be using maps from the literature and generating your own maps from public health data.

While we used storytelling and the role of CSs as important stakeholders in those narratives to support CSs’ sense of belonging (relatedness) and self-efficacy (competence), we also intended to support CSs’ autonomy through the user control and self-paced nature of the curriculum. In attending to CSs’ sense of belonging, self-efficacy and autonomy, we enhanced the alignment between our instructional design strategy and the key psychological needs described by the SDT framework (Ryan & Deci, 2017).

Interactive Embedded Assessments

Using the learning objectives as the blueprint, we created assessment items for each of the instructional videos within the CSCC. The creation and use of the assessment items were based on research and evidence on providing learners distributed practice experiences (Roediger & Karpicke, 2006) clearly aligned to learning objectives. For these assessments, we employed five multiple-choice questions on each practice assessment that combined recognition, analysis and evaluation of the content covered in each video. The combination of different types of questions was used to create a balanced learning experience where learners could contextualize the content and develop higher-order learning outcomes (Buchanan Hill, 2016). To illustrate how these types of questions were implemented, we provide a few examples from the assessment related to the instructional video “Causes of Cancer” in Table 2 with the correct response presented in bold.

Table 2 Example assessment items from embedded assessments
Full size table

To support CSs’ conceptual learning (Finn et al., 2018) and transfer to novel contexts, all assessment items had elaborated feedback for both correct and incorrect responses. Since feedback is a powerful learning aid, we chose to incorporate the feedback messages for all of the multiple-choice questions (Shute, 2008). All assessments were carefully reviewed by the team to ensure best practices in multiple-choice item-writing (Miller et al., 2013) and were also reviewed by the SMEs to ensure accuracy of the content.

Spotlights on Citizen Scientists

We intentionally included videos of the CSs sharing their motivation, personal experiences and how those aspects influenced their interest in cancer-related research. The CS spotlight videos create authenticity and relevance for the learners and can instill in them a sense of belonging within the learning community (Balyasnikova & Gillard, 2018; Delmas, 2017). Similar to the approach used for the supplemental and instructional videos, the instructional design team met with each CS featured in the “Spotlight on Citizen Scientists” videos. The goal of those meetings was to provide background information related to the video production and prepare CSs for the general questions that would be asked to guide the conversational style intended for the video format. The questions asked included: a) what is your motivation for working on cancer-related research? b) can you please talk about your previous experiences with cancer and some lessons learned through those experiences? c) how do these experiences influence your work as a CS? In addition to these more generic questions, other questions were developed based on CSs’ initial narratives. For example, one CS described the importance of humanizing the patient. This prompted the design team to address this topic in the final video recording with the question “Based on your experience, why is it important to humanize the patient?” The “Spotlight on Citizen Scientists” videos were recorded using a video conference platform (Zoom) and ranged between 7:35 and 8:20 min in length. The edited final versions of the videos only show the CSs telling their stories and addressing related questions.

Patient Case Study Video Series

The patient case study video series was informed by the conceptual framework to a motivational and emotional account of the importance of cancer-related research and the critical role CSs play as advocates in this process. We intentionally captured the stories from a cancer survivor, his caregivers (i.e., parents) and the physician who provided the treatment to humanize the CS role in cancer-related research using the notion of storytelling. The development of the patient case study video series required two meetings for preparation: one with the patient’s physician and a second meeting with the physician, the patient and the patient’s parents. In the first meeting, the physician described the case from his perspective and how the patient was able to enroll in an experimental clinical trial. In the second meeting, the patient and his parents described the family’s journey from the initial onset of symptoms through the diagnosis and treatment of the patient’s cancer. These accounts helped the instructional design team identify key points that could be highlighted across six segments of the final version of the case study. Two segments addressed the patient’s perspectives and focused on his experience growing up with, receiving treatment for and becoming a survivor of cancer. Three segments focused on the physician’s narrative of the diagnosis, treatment plan, opportunity for an innovative cancer treatment and the collaboration between researchers and the patients’ health care providers and family. Finally, one segment focused on the patient’s caregivers and how they relied on extensive supportive systems (e.g., family, friends, doctors, nurses) to navigate the challenges imposed by a cancer diagnosis. The caregivers detailed how this support system was critical as they cared for their child and enrolled him in the clinical trial that ultimately contributed to long-term remission of the cancer and survival of the patient.

To ensure that the rights and well-being of the patient and his family were protected and that the project followed the most ethical approaches and guidelines, our team contacted the university’s cancer center office of communications at the planning stage of the project. As part of this process, the cancer center required the use of a consent form that the patient filled out with help from a trained communications person not involved in the development of the curriculum. The form was filled out prior to the shoot and before any identifying information about the participant was published in any format.

Development of Citizen Science Cancer Curriculum

The CSCC was developed as an online, self-paced resource to support the sharing and distribution of the relevant curriculum resources across medical institutions and to provide an engaging learning experience for the future CSs working at our institution. The materials can be adapted for different online platforms, including Learning Management Systems (LMSs) commonly used in institutions of higher education or on other Internet-based distribution systems. Licensed under Creative Commons, the CSCC supports open education and ensures that commercial entities do not monetize the resources.

The development of the resources closely mirrored the formative evaluation procedures used to ensure the highest quality outcomes, which used an iterative process of creation, formative evaluation and subsequent revision negotiated by the SME and the instructional design team based on data collected. We elected to build the curriculum within the existing web ecosystem at our institution, which uses the WordPress content management system for its web presence. One clear advantage to the WordPress platform is the wealth of plug-ins available to extend the functionality of the platform. We opted to employ the H5P plugin (H5P, 2021), which provides a suite of interactive experiences, such as the use of embedded assessments with multiple-choice questions and item-level feedback and auto-grading. All videos were recorded by our professional video staff at the institution, edited accordingly and uploaded to YouTube for distribution.

Formative Evaluation Method

Objectives for Evaluation

While the initial evaluation during the needs assessment stage served to support the rationale for the design and development of the CSCC curriculum, the formative evaluation based on qualitative and quantitative data was important to examine CSs’ affective and content-specific outcomes. This multi-method approach has been used in other design and development research studies (Richey & Klein, 2014). In terms of evaluation models, the formative evaluation reflects the reaction- and learning-level described in the Kirkpatrick Model, where learners’ reactions to the training (e.g., CS satisfaction with the instructional approach) and learning outcomes (e.g., CSs can identify the main causes of cancer) are assessed (Jones et al., 2018). The following objectives were considered based on the intent of the course and its design to support CSs’ motivation and learning outcomes: 1) the CSCC incorporates sound instructional design informed by the conceptual framework, 2) the CSCC supports the learning and application of cancer-related concepts among the CSs and 3) the CSs’ motivation to engage in cancer-related research projects is positively shaped by the design of the CSCC and its affordances. To assess the extent to which each of these objectives was achieved, we employed a range of formative evaluation techniques with the CSs to assess the resources within the CSCC using an iterative process of creation, evaluation and revision by the team and related stakeholders (e.g., SMEs). The last two levels of the Kirkpatrick Model (behavior- and results-levels), which assess how the training translated into behavioral changes (e.g., continuing engagement of CS with cancer research) and the outcome of the training (e.g., successful collaborations between CSs and cancer researchers), respectively, were not addressed in the present study.

Participants, Instruments and Procedures

The same seven CSs (2 Males, 5 Females) who participated in the needs assessment also participated in the design, development and formative evaluation (between five and seven participants in each session) of the CSCC. Their experience working as CSs ranged between one and five years. Following the research protocol approved by the university’s institutional review board (IRB), we collected data on CS’s learning performance and perceptions of the course and its learning materials via surveys in Qualtrics™. The formative evaluation of the instructional videos was conducted in nine separate instances, one per instructional video and included five-point Likert-type questions (1. Strongly Disagree to 5. Strongly Agree) to address the quality of assessment items (e.g., I think the assessment items were easy to understand) and instructional videos (e.g., The presenter did not use too much technical language or jargon). Standardized Cronbach’s alpha values for the quality of assessment items scores (α = 0.85) and for the quality of video scores (α = 0.83) indicate good internal consistency. Open-ended items for additional feedback were also used for additional insights.

To understand how the patient case study video series influenced CSs’ general interest/motivation, relatedness, competence/self-efficacy and autonomy as they related to cancer research, we asked four open-ended questions, which were presented in the survey along with the six videos associated with the patient case study:

  • Does this case study influence your motivation to work as a Citizen Scientist in cancer research? Please explain why or why not. (General interest/motivation)

  • Did the Patient Case Study help you relate to the challenges that patients, caregivers and physicians go through as they deal with cancer diagnosis, treatment and survivorship? Please explain why or why not. (Relatedness)

  • How does the Patient Case Study influence your thinking about how you can contribute to cancer research? Please explain. (Competence/self-efficacy)

  • Do you feel the Patient Case Study empowered you to advocate for the needs of all stakeholders involved in cancer research? Please explain why or why not. (Autonomy)

As the CSCC project was implemented in the 2020–21 academic year, the COVID pandemic shaped our approach to creating and evaluating the resources as we had to ensure the safety (e.g., adherence to Centers for Disease Control and Prevention [CDC] guidelines) of all stakeholders involved. Consequently, the videos and assessments were recorded and created out of order and evaluated asynchronously by the CSs using the Qualtrics survey platform.

Usability Expert Think Aloud

As the CSCC is an interactive resource with the option of both linear or non-linear navigation of the curriculum resources, we elected to seek the expertise of a user experience design professional to conduct think aloud analysis of the final CSCC prior to production release. The user experience design professional has more than 20-years of experience developing interactive learning resources. The user experience design expert carefully reviewed the full CSCC resource while using a think aloud approach as a video recording. The full think aloud video lasted approximately 30 min and attended to a range of instructional design, usability, navigation and accessibility thoughts on the CSCC. After receiving the video from the user experience design expert, the team carefully reviewed the points raised and made slight alterations (e.g., changed lettering, formatting and navigation) to the CSCC. This review served as the final check-point before the final production release of the CSCC.

Results of Formative Evaluation

Learning and Satisfaction

Quantitative and qualitative results from the formative evaluation sessions indicate that the curriculum and its learning materials where helpful to support CSs’ learning outcomes and motivation. More specifically, all instructional videos (IV) and related assessments presented over 85% of correct responses as shown in Fig. 3.

Fig. 3
figure 3

Overall percentage of correct responses per instructional video and related assessment

Full size image

Even when we consider specific items within each module, most items presented greater than 70% of correct responses. Only two instructional videos (Palliative Care and Clinical Trial Enrollment) had one item each with a lower percentage (see Fig. 4). These data suggest a clear alignment among the learning objectives, the instructional videos and the practice assessments.

Fig. 4
figure 4

Percentage of correct responses per question for each instructional video

Full size image

CSs’ learning performance in the assessment items indicates that they comprehended the topics with a high level of mastery and were able to apply the concepts covered in the modules to the criterion-referenced assessment items. This result aligns with CSs’ overall positive feedback of the assessment items and instructional videos (Fig. 5), which highlights the importance of involving CSs in the instructional design process from the onset of the project. Note, the quality of the assessment items and the videos were rated on a five-point scale.

Fig. 5
figure 5

Distribution of the mean scores across participants

Full size image

CSs’ critical reflections were also present in the data. Their reflections were particularly helpful to guide changes regarding content presentation and scaffolding. For example, for the Clinical Trial 101 module, a CS mentioned:

I also feel that more explanation should be given to Phase 0 as this is seldom mentioned, in studies, and how is it determined if the drug [works] or is useful to go one [sic] studying in phase 0???.

Similar critical feedback was provided for the assessment items, which was used to make useful changes to the language and presentation of these items. For example, for the module about the Importance of Social Determinants of Health (Instructional Video 6), a CS said:

The first assessment asking for regurgitate [sic] a statistic seemed a little unfair. I didn't think the point of the video was to remember if it was 17% or 71% but to recognize that there is a relationship.

The feedback provided during the formative evaluation were essential to guide revisions to assessment items and for the final recording of the instructional videos. The comments and suggestions provided by CSs were addressed in the slides and verbal remarks. For instance, in light of their comments about the phases of clinical trials, we added about 30 s of content, especially regarding phase 0, as it was essentially a passing mention earlier. Initially, we considered the information to be only a supplemental component, but since the CSs asked for more detail, we changed course and added more information in the presentation.

Instructional Videos

CSs’ feedback about the instructional videos indicate that their motivation was also supported by the interaction with the instructional materials. CSs were particularly appreciative of the attention presenters gave to their role as CSs, which fostered a sense of belonging and competence in their work as CSs, as described by one participant:

I felt valued as a Citizen Scientist and enthusiastic about my role in the research process.

However, when they did not feel the connection between the content being presented and their roles as CSs, they also voiced their concerns as illustrated by the feedback provided by another CS:

The information was clear, easy to follow and comprehensive. The actual ROLE of Citizen Scientists did not seem to be addressed.

Importantly, the CSs really appreciated the use of stories in the videos and assessment items to contextualize the concepts covered as seen in the following suggestions by a CS:

I think the scenario questions were particularly helpful.

Think there could be a few more scenarios to further ascertain understanding of the video material.

Patient Case Study

The feedback from CSs about the patient case study video series indicate an overall approval of the storytelling approach as an instructional strategy to support their motivation to work on cancer-related research. The feedback was mostly positive (83%) and outlined four emerging themes: a) importance of information and research, b) positive outcome, c) multiple stakeholder engagement and d) role of CSs (Table 3).

Table 3 Themes related to CSs’ feedback about the patient case study video series
Full size table

It is worth mentioning that there were a few negative (17%) and neutral (29%) types of comments as well. For example, one CS mentioned that despite the story’s happy ending, they did not perceive a connection with the role of CS in cancer research:

I do not see any empowerment related to my role as Citizen Scientist in this particular patient study!! I feel happy that the young man finally made a connection that worked to save his life.

The negative feedback suggests that the CSs felt entitled and comfortable to share their concerns when their expectations regarding the quality of the learning material and instructional approach were not met.

Discussion and Reflections

As this work is a design and development case study focused on the creation of an authentic learning resource to engage CSs in cancer-related research, our discussion is framed as context-specific lessons learned and discussion points (Richey & Klein, 2014). Creating effective learning materials and resources requires collaboration and expertise among many different relevant stakeholders (e.g., learners, instructional designers, SMEs) (Luo et al., 2020), often with different viewpoints and agendas. The creation of the CSCC was an especially unique design and development case study since the work was executed during a pandemic in which different constraints (e.g., CDC safety guidelines) influenced our overall approach. Because of the COVID pandemic and our concern for the safety of all stakeholders involved in the project, our team was forced to employ alternative methods for the development, evaluation and revision of the CSCC resources using a range of information and communication technologies. Though we faced many challenges and constraints in the creation of the CSCC, the overall project led to some reflections and discussion points for others working on similar projects to consider.

Starting with our needs assessment focused on both CSs and cancer researchers, we opted to maximize the mutual needs and interests of both parties involved with the CSCC. While the CSs are the intended learners for the CSCC, the cancer researchers were also primary stakeholders since a critical point of the program is to improve cancer-related research outcomes by involving well-prepared community members in these projects. The findings from the needs assessment formed the foundation of the curriculum topics (e.g., treatment of cancer) for the CSCC. Results from the needs assessment also guided our theory-driven instructional design approach focused on the motivational and emotional aspects of both researchers and CSs engaging in this work. Notably, the needs assessment was conducted entirely online through the use of information and communication technologies (ICTs) (e.g., Zoom) to collect the important information since this process began in the earlier stages of the COVID pandemic in 2020. While our team had to use a range of ICTs to carry out the needs assessment work, our appraisal of the quality of the information gained and the initial buy-in from the stakeholders (e.g., SMEs for the instructional videos) using this approach is that we did not have to sacrifice the quality of the process for the sake of adhering to safety protocols.

Our team’s experience with the original CS curriculum (Valle et al., 2018), shaped our engagement strategy for each of the stakeholders in the project. For instance, we knew from experience that creating effective instructional videos with the SMEs would require careful attention to the contents of the slides and scripts used in for the final recordings. One challenging aspect to working with the SMEs on the CSCC was the immense difficulty in scheduling the SMEs to record their videos in the production studio, as clinicians have often inflexible and variable schedules. As some of the SMEs were also surgeons, there were literal life-and-death circumstances that caused a need to reschedule. This reality forced the team to spend more time in the initial design stages to ensure consistency across the CSCC and to subsequently record the videos out of sequence. While this reality proved challenging, we quickly learned that producing the resources out of sequence would not hinder the project so long as appropriate design time was spent on the learning objectives, slides, scripts and assessments across the CSCC prior to full production. Though the SMEs were difficult to schedule, they were very excited about participating in the project to further refine the cancer-related research trajectory of the institution.

The engagement of multiple stakeholders (e.g., CSs, SMEs, video production team) from the inception of the project until its closing was essential for the alignment between the curriculum’s theoretical underpinnings and the learning materials developed. The validation of our instructional design approach came in the form of learning outcomes and CSs’ feedback on how they felt valued and how the storytelling component supported their interest (attention to the importance of cancer-related research and access to information), relatedness (emotional involvement, sympathy for the struggles of caregivers and patients) and sense of competence to contribute to cancer-related research.

Taken together, the processes and outcomes of this design and development case study indicate that the integration of learning sciences (e.g., SDT) into the instructional design practice to attend to learners’ psychological needs and well-being is a viable solution for successful curriculum development, especially around sensitive topics such as cancer. The robust needs assessment was essential to support the negotiation of differing priorities toward a common goal: the successful engagement of CS in cancer-related research projects. Future studies in similar contexts could strive to include summative evaluations and assessment of behavioral changes among learners. For example, questions such as “Do the learning materials support the long-term engagement of CSs in cancer research?” or “Does the use of storytelling facilitate long-term knowledge retention?” could be addressed.

Closing Remarks

While community stakeholder engagement is a critical component of clinical and translational research processes to ensure alignment between research practices and the need of targeted populations (Stallings et al., 2019), there is limited literature to guide institutions and health professionals on how to support the training of community stakeholders so they can become active members of the research team (Valle et al., 2018). The purpose of this design and development case study was to assess how the design and development of a curriculum informed by SDT, which used storytelling as an instructional strategy, influenced CSs’ motivation and learning outcomes in the domain of cancer research. The results indicate that the instructional design process (Morrison et al., 2019) was key for the creation of quality assessment items and instructional videos based on their alignment with the instructional goals set in the initial stage of the project. Furthermore, CSs’ feedback indicate that the CSCC curriculum supported CSs’ learning and motivation to engage in cancer-related research projects. Finally, the use of storytelling was essential to create interest in the topic and support learners’ psychological needs for relatedness and competence in the context of the CS program. Although there are some context-specific aspects that may limit the generalization of this case (Richey & Klein, 2014), we hope the detailed descriptions and reflections will provide invaluable insights into how instructional design approaches and the use of storytelling as an instructional strategy hold potential for supporting similar community stakeholder engagement initiatives.