INTRODUCTION

In 2020, lung cancer accounted for 22.4% of all U.S. cancer mortalities.1, 2 The National Lung Screening Trial demonstrated that annual LDCT screening has the potential to reduce lung cancer mortality by 20% at 3 years3 and 16% at 7 years.4 The Centers for Medicare & Medicaid Services (CMS) current eligibility criteria for LDCT are individuals 55–77 years old who have at least a 30 pack-year history and former smokers with fewer than 15 years of abstinence before screening.5 In 2015, CMS required a counseling and shared decision-making (SDM) visit before LDCT to receive reimbursement. This documented visit addresses patient preferences, the importance of adherence to annual LDCT, smoking abstinence and cessation interventions, benefits and harms of screening, follow-up diagnostics, over-diagnosis, false-positive rate, and total radiation exposure.5 While roughly nine million Americans meet the CMS criteria, many do not pursue LCS, as stigma, mistrust, socioeconomic disparities, and unfavorable harm-benefit perception act as barriers.6,7,8,9,10 Studies suggest that providers are likewise reluctant to adopt the required SDM before LDCT due to low reimbursement, time demands, limited training, and unfamiliarity with decision aids.11,12,13,14,15 Consequently, SDM uptake has been low for both patients and providers.10

Decision aids—educational tools available to patients to balance information asymmetry and align patients’ preferences—are required for SDM by the mandate, though it does not obligate the use of one specific instrument.16 University of Minnesota Health, Siteman Cancer Center, Mississauga Halton Central West Regional Cancer Program, Salem Hospital, and University of Colorado Hospital providers recommend web-based lung cancer risk calculators (LCRCs) to SDM patients per mandate requirements17,18,19,20,21. While certain institutions’ websites refer patients to a specific LCRC, most LCRCs exist on the web independently. These LCRCs use risk prediction models to deliver a risk depiction (i.e., quantitative and/or qualitative representations of cancer probability) and screening guidance. Risk prediction models estimate the probability of developing lung cancer within a specified time-window based on a clinical cohort representative of the CMS-eligible population, health indicators, and statistical formula.22, 23 In a comparison of nine prominent lung cancer risk models, Katki et al. identified four consistently high-performing models: the Bach model, the PLCO model, LCDRAT, and the Lung Cancer Risk Assessment Tool.37 These models provide physicians and their patients’ seeking additional information about lung cancer probability based on the patients’ individual risk factors.

According to Kuhlthau’s Information-Seeking Process (ISP) model, individuals have limited capacity to convert new information into knowledge, and thus purposefully construct meaning by selectively attending to information that connects with pre-existing knowledge.24 As inconsistencies and incompatibilities in information are confronted, doubt in the new information arises.25 Studies on the effectiveness of LCRCs following SDM illustrate contradictions, showing that they lead to either negative26, 27 or positive16, 28, 29 patient outcomes, exacerbating these problematic inconsistencies. To overcome converting new information to knowledge issues addressed in the ISP model, web-based LCRCs may act as an entry way to SDM as a decision-aid to discuss the benefits and harms between physicians and patients. This SDM opportunity is often missed due to low uptake by primary care providers; individuals who would benefit from LCS are not receiving information to make educated, value-driven decisions.30, 31 Instead of talking to a provider, 39% of Americans begin their search for health information online, utilizing both institutional and external websites.30, 32 However, the accuracy and genre-specific interfaces of health information websites across the web are inconsistent, raising concerns about the quality of information and the user’s ability to critically evaluate online resources, especially as the CMS-eligible population is more likely to experience lower online health literacy.33,34,35,36

Therefore, the depiction of lung cancer risk should be accurate, consistent, and easily understood.

The variability of web-based LCRCs may be providing varying information to users which may be in turn impacting perceptions of the necessity of SDM uptake. The objective of the study is to inspect the heterogeneity of web-based LCRCs via their inputs, outputs, and risk depiction characteristics and discuss practical implications.

DESIGN

Search and Selection Criteria

A systematic Internet search was conducted in June 2019 to locate and identify LCRCs available to and trafficked by the public. The search was conducted using Google, Bing, and Yahoo search engines and scholar indexes services, including Google Scholar, PubMed, Medline, and EBSCO. Keywords included [(“Lung cancer calculator”) or (“Lung cancer risk”) or (“Lung cancer risk assessment”)]. LCRCs that met the following criteria were included: (1) appeared in the top five pages of inter-browser organic search results, (2) contained an interactive LCRC, (3) included certain input variables (e.g., age, sex, race/ethnicity, BMI). LCRCs that requested inputs implying past encounters, such as previous diagnoses of lung nodules, were excluded, as the study focused on patients new to the LCS process. Time-windows were not considered during the review process.

Evaluation of Risk Calculators

The risk prediction models used for identified LCRCs were determined either directly from the website or by contacting the calculator authors via email, telephone, or site-based communication. Many websites featured literature about the LCRC’s particular risk prediction model formulas used to generate their risk estimate.23, 38,39,40,41 Categorical questions and answers were inspected and standardized across the analyzed LCRCs. Input factors and the subsequent outputs are contingent on the calculator’s risk prediction model, as each model’s respective formula uses different combinations of variables.

Example Patients

To evaluate both the variance of LCRC results and capture the considerable breadth and diversity of the potential CMS-eligible patient sample,42 16 hypothetical patients were generated using R, each representing a variation of the most common characteristics evaluated (see Table 1). The number of hypothetical patients was chosen to allow for a diversity of patient characteristics to compare the risk predictions of LCRCs through a diverse and randomized patient sample while allowing for a quick collection of the data as each hypothetical patient had to be entered into each LCRC individually. The hypothetical patients’ randomly assigned input variables were those used in the well-calibrated PLCO model (Table 1).37 Since many questions were phrased differently or were hidden from the user depending on the calculator’s conditional branching, the study categorized each calculator’s questions based on the health indicator in consideration (Table 2).

Table 1 Sample patient characteristics
Full size table
Table 2 Risk Calculator Input Characteristics
Full size table

Statistical Analysis

To generate 16 hypothetical patients, the predominant input variables used by the LCRCs were given ranges from which those attributes could be randomly assigned to limit potential bias. The full list of variables generated was as follows: age, sex, race/ethnicity, BMI, height (inches), weight (pounds), current smoking status, smoking duration in pack-years, education, personal or family history of cancer, and exposure. Exposure is a Yes/No category to include the personal/environmental risk factors considered by the LCRCs and, for all but one calculator (mensxmachina.org), would stand for pulmonary disease and/or asbestos exposure. The BMI range was 18–36 to allow for a range of weights to be randomly assigned to the hypothetical sample. The height of the sample patients was randomly selected from the 10–90% range of height in inches rounded to the nearest whole number, both for males (65–73 inches) and females (60–67 inches). The pack-years of the hypothetical patients ranged from 30 to 40 pack-years in integer years.

For the statistical analysis, LCRC results were separated by respective time-window (1, 5, 6, 9, and 16 years). Statistical analyses were conducted for time-windows that had at least three LCRCs (5 and 6 years) to compute the sample variance and sample standard deviation for each patient’s output. Additionally, for each patient, the calculator results were compared to the set “gold standard” PLCO model within the same time-window.37 Statistical analysis in the case of a set standard was calculated by setting the assumed population mean to the result of the set standard and calculating the sample variance and sample standard deviation under this assumption.

RESULTS

Lung Cancer Risk Calculators

From the inclusion criteria, 11 calculators were initially identified (n = 11) (Table 3). One calculator containing questions regarding previously diagnosed lung nodules was excluded, as this implies the patient has already received LCS and may or may not have participated in SDM (n = 1). The sites that use the “gold standard” PLCO model include shouldiscreen.com, analysistools.nci.nih.gov, merckmanuals.com, and aats.org.

Table 3 Name of site and model used
Full size table

Differences in Risk Calculator Inputs and Risk Factors

All selected calculators have input variables that fall into four predictive categories: demographic factors, lung cancer history, smoking status, and personal/environmental factors (Table 2). Two LCRCs considered whether an individual has held residence in a city with a population >100,000 for at least 10 years, to account for environmental carcinogens in cities of this size and the time necessary for these pollutants to affect the individual’s health.43, 44 Personal/environmental factors included health history and physical surrounding characteristics.

Several inputs are shared by a majority of the LCRCs: age (n = 10), duration of smoking habit (n = 10), current smoking status (n = 7), average cigarettes per day (n = 9), sex (n = 9), family history (n = 7), personal history (n = 5), asbestos exposure (n = 5), and height and weight (n = 5). These similarities are beneficial for potential SDM patients as the first three inputs listed above directly relate to the CMS criteria for recommended annual LDCT.

Differences in Risk Calculator Outputs and Risk Depiction

The analyzed calculators also vary in output characteristics (Table 4) and risk depiction for hypothetical patients (Table 5). Output characteristics were grouped into categories to illustrate calculator heterogeneity: screening eligibility, quantitative risk depiction, qualitative risk depiction, and other miscellaneous outputs. For example, the University of Michigan’s shouldiscreen.com provides a percentage risk of developing lung cancer within 6 years, as well as a recommendation for screening. Estimates of the probability of lung cancer development vary in length of time for development from 1 to 16 years.

Table 4 Output characteristics and risk depiction
Full size table
Table 5 Calculator results for sample patients
Full size table

Hypothetical Patient Results

There were 13 total instances across 10 of the 16 hypothetical patients in which the sample standard error exceeded the mean risk percentage across all general samples and set standard calculations (Table 6). The patients with significant variations did not share any consistent traits, as they ranged across age (59–76), BMI (20–35), and pack-year (31–40); they consisted of both sexes (M=6, F=4), all considered race categories (White=2, Black=2, AIAN=3, NHPI=3), and across personal history (Y=4, N=6), family history (Y=3, N=7), and exposure (Y=3, N=7), and every education level almost equally. Directly comparing the PLCO LCRCs used as set standards for the 6-year time-window statistical analysis (shouldiscreen.com, mereckmanuals.com) has a 4.83% average difference, with a 3.56% median. The largest measured difference is 16.49% for patient 8, and the smallest difference is 0.01% for patient 2. There were four instances where the PLCO LCRC differences were above 5% for patients 1, 8, 9, and 11.

Table 6 Statistical Analysis
Full size table

DISCUSSION

The Status of Web-Based Lung Cancer Risk Calculators

This study found non-significant variations in risk prediction across the LCRCs for six of the 16 hypothetical patients, but for the remaining ten hypothetical patients, there was a large discrepancy in risk prediction. In one case, percentage risk ranged from 2.69 to 29.49%, which merited closer review of each calculator’s inputs.

This significant range in risk percentages is concerning, as many of the risk models used by web-based LCRCs are validated by either a testing subset of the original study data or additional external datasets.21, 40, 41, 45 The patient sample in the datasets used to create and validate these models fall almost entirely within the CMS criteria for annual LDCT, and when compared directly across identical time-windows, there are significant variations in risk prediction across all ranges and categories considered for the hypothetical patients.

In Table 2, there are ten smoking-related questions that vary among the calculators. Lung cancer risk varies widely between smokers due to individual health indicators, but the outputs do not reliably reflect this.45, 46 This study likewise finds heterogeneity in the risk depiction for each hypothetical patient. For example, the depiction of the additional mortality risk metric and the qualitative lung cancer risk depiction (e.g., thermometer graph on siteman.wustl.edu, low/intermediate/high-risk thresholds on mycanceriq.ca) are likely to confuse patients seeking accuracy and clarity for their understanding and decision-making.

Furthermore, this study found six instances where the standard error of the general sample exceeded the average risk percentage of the sample for both the 5-year and 6-year time-windows combined and 13 total instances of the sample standard error exceeding the average risk value for both general samples calculations and set standard calculations. The heterogeneity in these numerical results implies that prospective or current LCS patients may arrive at different conclusions regarding whether to go for a check-up and/or undergo LDCT depending on which LCRC they come across.

Differences in Risk Prediction Models

The variation in risk prediction returned by the LCRCs and, by extension, published risk prediction models are partly attributable to the various time-windows considered by each model37. Patients and providers must be cognizant of some models’ tendency to overestimate risk before interpreting these predictions or referring patients to them as a decision aid.37

Implications on the Shared Decision-making Process

Provider/patient decisions and beliefs may potentially be impacted by the differences between the tested calculators. If a user is searching for web-based LCRCs before SDM-LDCT, the variability among calculators may affect their initial decision to see a provider.47 Following the ISP model, imprecise, conflicting, and esoteric information increases uncertainty and doubt, especially in individuals with lower levels of health literacy.48 The heterogeneity of inputs and outputs may compromise decision-making during the SDM process, leading to underutilization.7,8,9,10 Identical users trying merckmanuals.com and omnicalculator.com, for example, may find their respective risk percentages to be disconcertingly high or comfortingly low; subsequently, one user may seek LCS while the other may not.47

For the provider recommending online calculators, there is the dual responsibility of ensuring calculator quality and maintaining awareness of problematic inconsistencies. Reluctance to recommend online resources, unfamiliarity with the websites, low inter-rater reliability, and confusion regarding CMS guidelines frustrate this task and lead to low provider uptake.49, 50 As a result, patients receive mixed messages and may attribute such conflict to provider bias or incompetence.51 These barriers at both the provider and patient-level reduce the likelihood of LCS initiation and adherence and corrode the patient-provider SDM balance.52 As trust is integral to patient vulnerability, its loss can result in disappointment that prejudices future encounters and reduces patient self-efficacy.48, 51 This study demonstrates a need for web-based LCRCs to standardize their risk prediction calculation and presentation to the most well-calibrated model in order to best serve eligible patients.

LIMITATIONS

These findings should be understood through several limitations. The study analyzed different LCRCs through hypothetical patients with characteristics of the eligible population. As the possible range of characteristics and experiences is limitless, these 16 simulated individuals may not fully replicate the population of interest. Each website utilized different combinations of risk prediction models, user interface designs, input responses, and output characteristics, reflecting the variability of the online landscape. These differences may limit the generalizability of these findings to other risk assessment tools.

CONCLUSION

This study presents inconsistencies in predictive performance and risk depiction between web-based LCRCs. Although each calculator uses risk prediction models based on clinical populations, they display user risk in incongruous and non-standardized manners. Moreover, the variability of these calculators may impact users’ assumptions and beliefs about the accuracy of web-based health information and the reliability of provider recommendations. Standardizing web-based LCRCs so that they are both reliable pre-LCS tools and decision aids is critical to building trust in online health resources, providing users with useful information, and facilitating the lung cancer screening SDM process.