Challenge Statement

In design-oriented biomedical engineering courses, some instructors have adopted need-driven methods like the biodesign innovation process for teaching health technology innovation. At the heart of this type of approach is the need statement, which describes in one sentence the problem or health-related dilemma that requires attention, the population most affected by the problem, and the targeted change in outcome that is most vital and against which all potential solutions to the need will eventually be evaluated [1]. The need statement is dynamic, reflecting at any given moment a student team’s hypothesis about the most fruitful direction for their project. Students learn to scope and refine their need statement over time based on increasingly in-depth primary and secondary research.

Directional information about how to write a need statement, including the standard format it takes and common pitfalls to avoid in drafting one, are available in resources, like the Biodesign [1] textbook and the online Student Guide to Biodesign [2]. However, while rubrics are a key tool in biomedical engineering education to assess a range of outcomes and assignments [3], we were unaware of any comprehensive rubric for helping instructors systematically evaluate student need statements or directly aiding students in learning how to write them. One study provided students with a series of worksheets to lead them in the construction of a need statement, but the emphasis of this effort was on assessing whether they had identified a “problem worth solving” according to criteria set forth by the university office of technology transfer rather than evaluating the need statement itself [4].

The education community generally defines a rubric as outlining the expectations for an assignment by providing (1) the criteria that will be used for assessment; (2) detailed descriptions of different quality levels for each criterion (e.g., excellent to poor); and (3) a rating scale that can be used to provide a score for each criterion [5]. Multiple studies find that undergraduate and graduate students consider rubrics valuable to their learning because they clarify key targets to focus on for an assignment; allow for self-evaluation and improvement before assignment submission; and align the expectations of students and instructors around quality standards and corresponding grades [5, 6]. As for instructors, the literature suggests that they value rubrics for the role they play in enabling expeditious, objective, and accurate grade assignment [5].

Motivated by the potential to enhance the student learning experience, we decided to create a rubric for evaluating need statements. We deemed this work especially important because (1) need statements play such a foundational role in setting the direction for student projects and (2) need statement development has been described as more “art” than science [1]. With regard to this second point, conversations with individuals teaching across our undergraduate, graduate, and post-graduate programs quickly revealed that our faculty did not share a common point of view on what constitutes a high-quality need statement. Even those teaching in the same courses had differing factors they looked for when assessing a need statement. As a result, we determined that a clear and objective rubric would be a useful tool to facilitate greater alignment across our faculty as they interact with and provide feedback to students on their needs. It also could serve as a learning aid that students use to help demystify the “art” of need statement creation.

Novel Initiative

We took an iterative approach, using input from approximately a dozen faculty members who teach undergraduate, graduate, and fellowship-level health technology design courses/programs at our university, to define assessment criteria, performance levels, and a scoring construct for need statement evaluation. This work resulted in a set of six objective and semi-objective factors that any instructor could apply when evaluating a need statement. These six criteria primarily assess the construction of a need statement, with a focus on the presence of a problem, population, and outcome, as well as the alignment and interplay between those aspects (see Table 1). The construct we propose explicitly separates need statement construction from need statement content (e.g., whether the need statement will be compelling to key stakeholders in the need area or if it represents a promising innovation project) because content assessment is far more subjective and accurate judgment often requires comprehensive expertise in the clinical need area itself. Considering the number of different clinical specialties addressed in any given course or program, most faculty members will not have sufficient depth of knowledge to fully assess the content of each need. The proposed approach uses the rubric for need statement construction to assign scores/grades. With regard to content, instructors can gauge and comment on the level of “fluency” the team has achieved in the need area, encourage the students to share what they believe to be unique insights uncovered through their research, and involve relevant subject matter experts to lend their input to the evaluation of content.

Table 1 Rubric for evaluating need statement construction
Full size table

We introduced the initial version of the rubric to students participating in a 3-week intersession course that provides a total of 12 rising sophomores per year with the opportunity to practice the earliest stages of the biodesign innovation process. Specifically, they receive didactic instruction on performing clinical observations and then gain access to the hospital and clinics to conduct clinical shadowing in pairs over 3 days. Then, in parallel with lectures on each of the following topics, individual students draft preliminary need statements from their observations, perform research in each need area, and use what they have learned to scope and refine their need statements before filtering to a lead project. For the fall 2023 offering of the course, we shared the rubric with all 12 students and provided it to the instructors to use as a guide when coaching the students through need statement development and refinement.

After the course concluded, our course manager compiled and de-identified the final need statements for each student’s top project in the current (2023) cohort, along with the final need statements from students enrolled in the same course in fall 2021 and 2022 (prior to the rubric’s development and use). Students in all 3 years of the class had consented to have their work anonymized and reviewed for the purposes of educational research and publication (under IRB approval, protocol ID 56713). The de-identified need statements were “shuffled” across years and made available to three instructors who independently assigned retrospective scores to each one using the rubric. The three instructors had deep familiarity with need statements, as well as experience teaching in undergraduate health technology design courses.

Our assumption was that the need statements from 2023 would score better because the rubric was available to students and the instructors during the course (note that the instructors were the same and student demographics were similar across all 3 years the class was offered). However, data analysis of the 12 final student need statements from 2021, 2022, and 2023 (36 need statements in total) revealed that the highest scoring need statements were split across all 3 class years (within one standard deviation of each other) as shown in Table 2. A Wilcoxon sum ranks test confirmed that there was no statistically significant difference among the cohorts. The results also were characterized by a high level of inter-rater variability in the scores assigned to each need statement as shown in Table 3.

Table 2 Comparison of final 12 need statements from 2023, 2022, and 2021
Full size table
Table 3 Alignment of reviewer scores
Full size table

Reflection

While these results do not support the assumption that using the rubric would directly improve student-created need statements, we are sharing this work because we believe there is a great deal to be learned from this first attempt at rubric development, implementation, and evaluation. We hope that other instructors who teach need statement development in their design-oriented biomedical engineering courses will find the rubric (Table 1) interesting and adapt and improve upon it for their own use. We also would like to highlight the following lessons.

First, this experiment convinced us of the value that rubrics can provide in aiding student learning. Anecdotally, students told us that the rubric clarified, in more concrete terms than the didactic instruction and need statement examples they were shown in lectures and readings, what constituted a well-constructed need statement. Additionally, as the literature suggested, they liked that they could use the rubric to “self-check” and improve their work. Table 4 shows the first draft and final need statements of two students from the 2023 cohort as subjective examples of how students applied the criteria in the rubric to produce stronger need statements as they evolved from “first draft” to “final.”

Table 4 Examples of how student need statements improved against rubric criteria
Full size table

Second, the teaching team felt more aligned as a result of having the rubric to collectively refer to and we believe that it helped us deliver a more unified approach when individually coaching students 1:1 to fine-tune their needs. We also found the distinction between need statement construction and content to be helpful. We often explain to students that there is no “right answer” when considering unmet clinical needs. Separating need statement construction from content allowed us to firmly hold students accountable for gaining skills in constructing need statements while taking a more exploratory, co-learning approach to need statement content.

Third, the experience uncovered improvements we can make to the rubric itself. For example: (1) The performance levels for the first criterion (“Does the need statement include a problem, population, and outcome, with each element clearly and singularly articulated?”) focus on whether or not the problem, population, and outcome are all clearly and singularly articulated, but they do not take into account whether a student may have failed to include one of these important parts, which is a common problem with need statement construction. (2) The way that criterion 4 is stated (“Is there a temporal and causal linkage between the problem and outcome?”) refers to the connect between the problem and the outcome, but more accurately should reference the link between the problem if solved and the desired outcome. (3) Criterion 5 (“Is the outcome objectively measurable within a timeframe that’s reasonable given the magnitude of the problem?”) has a compound focus—objective measurability and the required timeframe to perform the measurement. These may be better evaluated by two separate criteria. We are eager to revise the rubric to address these and other insights gleaned through trialing the preliminary version.

Fourth, we have the opportunity to proactively develop a robust study protocol that will strengthen our understanding of the rubric’s effectiveness. This approach should include a clear methodology for introducing and testing the rubric with students, evaluating the work they produce using the tool, and capturing their perceptions of its value through surveys. Our approach also should be informed by best practices in rubric development and implementation. For example, a frequent measure of rubric effectiveness is the consistency of grading between different raters [6]. However, as shown in Table 3, we had high inter-rater variability in the scores assigned to each need statement. A debrief with the members of the teaching team who evaluated the need statements revealed inconsistencies in how we understood the criteria and performance levels and, accordingly, how we assigned our scores. An effective rubric depends on clear, understandable language that is consistently interpreted and applied by raters and students alike [5], so we should seek input from students and faculty when updating the rubric and then allocate time to training raters on the revised rubric, using sample need statements to achieve greater score alignment, before initiating a formal study.

In addition to these improvements, we see other interesting opportunities for future investigation. For instance, given advances in generative AI, we have had discussions about what role this technology could play in applying rubrics to student work. Designing a study to compare and contrast how instructors and generative AI evaluate a common set of need statements against a rubric could be a fascinating experiment in further exploring inter-rater variability, as well as the capabilities of tools, such as ChatGPT (OpenAI, San Francisco, CA). Another idea is to explore one of the most common criticisms of rubrics that they can promote “instrumentalism” if students only do the minimum necessary to receive their desired grade rather than completing more thoughtful and creative work [7]. As we design future studies, we hope to do so in a way that provides insights into this possibility, perhaps through the use of a control group to enable direct comparisons. Finally, we see great opportunity to develop additional rubrics to help clarify other key tools in the biodesign innovation process, such as need criteria (the requirements students create as a culmination of their need research to enable ideating and screening solution concepts). As with need statements, we are not aware of any objective rubric for guiding the development or evaluation of need criteria. We look forward to sharing the outcomes of future studies as our efforts continue.