Key message

  1. 1.

    Current COVID-19 case statistics reported to the public by Canadian news outlets and governmental websites do not abide by epidemiological reporting standards and show important data gaps such as lack of COVID-19 case data on race and ethnicity.

  2. 2.

    Population-wide representative COVID-19 testing should be implemented to allow for accurate monitoring of the scale and dynamics of the COVID-19 epidemic in Canada.

  3. 3.

    As currently used indicators for monitoring the pandemic (e.g., COVID-19 case statistics) are surrogate measures prone to large imprecision, more focus should be given to resource-centric measures such as required hospitalizations and occupation of intensive care unit beds in relation to known capacities.

Introduction

The coronavirus (COVID-19) pandemic presents unprecedented challenges. Widespread cases of COVID-19 and related preventive measures recommended by health authorities and implemented by governments have continued to impact societies worldwide. As this unparalleled and ongoing situation evolves in Canada, provincial and territorial governments continue to make decisions that affect people’s day-to-day lives and critical national infrastructures. Thus, a critical appraisal of the foundation of decision and policy-making is essential: the epidemiological COVID-19 data collected and reported to the public. Several authors have already commented on challenges related to accurate COVID-19 disease surveillance and modelling of the epidemic situation [1,2,3].

Minimum requirements for robust and practically relevant inference from population disease data have long been established in the epidemiological literature:

  • Consistent case definition: consistent case definition and unambiguous application of a clinically meaningful diagnostic criterion of the disease in the target population [4,5,6],

  • Large and representative samples: use of sufficiently large, representative and repeated samples to monitor the prevalence, incidence, and spatio-temporal spread of the disease [2, 7, 8], and

  • Use of appropriate denominators: appropriate standardization and representation of disease cases to enable an unbiased evaluation of the epidemic over time and across geographical regions or subpopulations [9, 10].

During the ongoing COVID-19 pandemic, we have observed with utmost concern that none of these criteria have been met for epidemiological data routinely presented in Canadian public communications by official news outlets and government bodies. Researchers raised similar concerns about the quality of reporting about the 2014 Ebola epidemic in West Africa [11]. A systematic analysis of 69 Ebola epidemic reports found that only 70% included case definitions and 84% included proportions of patient outcomes such as hospitalizations, mortality, and intensive care unit admittance [12]. These findings draw attention to a serious weakness: that use of appropriate epidemiological standards continues to be a challenge in disease reporting. The systematic analysis included only articles published in scientific journals and no reports published by news outlets or governmental bodies. Some articles [11, 13, 14] did assess news outlet and social media reporting of the Ebola epidemic, but primarily looked at the impact of sensational reporting, with less focus on epidemiological reporting standards. The authors concluded that the way in which infectious disease data are reported can influence the public perception of risk associated with a disease.

Given the central role of public disease surveillance in a pandemic, we sought to appraise the COVID-19 data reporting of the Canadian government (federal, provincial, and territorial) and major Canadian news outlets. We assessed whether routinely reported COVID-19 data are accurately informing the general public about the epidemiological situation. We provide an overview of the importance of epidemiological disease reporting standards and the implications of not abiding by them, substantiated by the news and government data we collected.

Overview of epidemiological reporting guidelines

The importance of reporting case statistics with appropriate denominators

Proportions and rates are fundamental concepts in descriptive statistics and epidemiology, because putting observations (such as case counts) in relation to time and populations-at-risk enables a fair comparative assessment of the relevance and dynamics of the problem [2]. Epidemiologists commonly use two indices to describe the presence and emergence of a disease: prevalence and incidence. The former describes the relative frequency of a condition in a population at a single point in time or during a specific period of time [15,16,17,18,19,20]. The latter captures the rate of emerging cases in a specific population, typically reported as the number of new cases per total observation time, for example, cases per 100,000 person-years [15,16,17,18,19,20].

In the definitions of both indices, ‘population’ refers to a distinct group of individuals who are, in general, ‘at risk’ of developing the condition of interest, the population-at-risk [18,19,20]. In practice, however, the population-at-risk may take into account individuals who actually have zero probability of acquiring the condition under study. For instance, when estimating the prevalence or incidence of shingles in a population, the population-at-risk is typically defined as ‘all adults’, disregarding that a shingles infection is predicated upon a previous varicella-zoster virus infection, potentially miscounting individuals who have not had such an infection. Although such imprecisions may affect the overall accuracy of the estimated indices, one can typically assume consistency of these errors over time, allowing for informative monitoring of changes of the population disease burden over time.

The curse of dynamically changing invisible denominators

In the context of disease surveillance, the population-at-risk is not only defined by potentially susceptible individuals, but more strictly by individuals-at-risk who actually have the opportunity to undergo diagnostic testing [19]. In settings where a large proportion of infected individuals remain asymptomatic throughout the course of the disease and testing is primarily available to symptomatic individuals or selected subpopulations (such as close contacts of individuals who tested positive or health professionals potentially exposed to infected individuals)—neither the numerator (case count) nor denominator (population-at-risk to be diagnosed) are accurate. They do not reflect the actual population quantities of interest, rendering prevalence and incidence estimates invalid [17]. In situations where specific sub-populations (such as health workers at a particular site) undergo routine testing, estimates may be useful for monitoring this sub-population.

One might argue that counting confirmed cases in these selected populations suffices to approximate the infamous “[epidemic] curve [to be flattened]”. This approach is, however, problematic for several reasons. First, the number of positive test results strictly depends on the availability of testing and the number of tests conducted in a specific region and population at a given time or time period. These capacities are largely time-dynamic and selection criteria for testing change over time, often in response to emerging evidence on local outbreaks or potential mass exposure to infection. Such events inevitably lead to dynamic changes in the population-at-risk with access to diagnostic tests; hence they lead to unpredictable variations of cases expected over time.

Importance of representative testing

Performing diagnostic testing predominately in symptomatic individuals and non-representative subpopulations does not allow for estimation of the prevalence of currently infectious individuals who pose immediate risk to others. Nor does it allow for estimation of the proportion of infected individuals who show few or no symptoms, an important index for understanding the utility of symptom (self-) screening or monitoring implemented as one of Canada’s COVID-19 pandemic response criteria [21]. Random (household) sampling enables an accurate representation of the population and its manifold characteristics when determining the epidemic situation in a population. It may be surprising to the reader, but even with a size n = 500 random samples (in a specific neighbourhood, for example), an estimated proportion (the prevalence of COVID-19 positive persons) would yield a half-width of less than 5% for the respectively associated 95% confidence interval.

Several types of COVID-19 tests are available across Canada. Canadian health authorities employ molecular polymerase chain reaction (PCR) tests to detect the presence of the COVID-19 viral DNA via nose swab, throat swab, or saliva sample [22]. For rapid screening of COVID-19 cases, Canada uses point-of-care tests such as rapid antigen tests. Administration of the latter is more rapid, but the results are less accurate than PCR tests in detecting exposure to COVID-19. Though point-of-care tests may be more accurate for detection of transmissible disease [23]. To identify a previous COVID-19 infection, Canadian health authorities use antibody (serology) tests, but these offer limited diagnostic value [22]. Eligibility for PCR COVID-19 testing varies across Canada’s geographical regions (provinces and territories) and has, initially depended on the presence of symptoms. Availability of the types of COVID-19 tests also varies across regions; some provincial or territorial websites do not explicitly state the type of COVID-19 tests used. Based on a review of governmental websites (accessed on 5 August 2021), we found that only 7 out of 13 provincial and territorial websites explicitly stated the types of COVID-19 testing available. We also found that news reports frequently do not differentiate amongst types of tests. If symptoms that qualify individuals to undergo diagnostic testing are not disease specific and are associated with other conditions prevalent in the population, e.g., influenza, seasonal variations in the manifestation of these alternate conditions are also important for determining which populations to test.

Methods

The World Health Organization’s definition of health, as “the state of complete physical, mental, and social well-being” [24]. Thus, we believe that an assessment of the health-based criteria (such as age, gender) and the disparities in the social determinants of health is essential to a full and meaningful discussion about health, particularly in this unprecedented pandemic.

We sought to appraise the COVID-19 reporting of official governmental websites of the provincial and territorial health institutions and the top 15 Canadian news outlets according to a large international media outlet database (www.allyoucanread.com) in Canada over time from 2020 to 2021: on 28 April, 2 June, 29 June, 15 September 2020; 15 January, and 19 August 2021. We chose to appraise the news outlets reporting of COVID-19 because, according to framing theory, the way the media frames (or reports) an issue can influence individuals’ perceptions of it, and affect their attitudes or behaviours [25]. As observed during other infectious disease reporting such as with Ebola [13], information reported by news outlets may influence actions (or inactions) of individuals in society to protect public health.

A 2001 guideline from the US Centers for Disease Control and Prevention (CDC) on surveillance [26] provides general recommendations on quality assurance of data reported from a surveillance system. These recommendations confirm the importance of epidemiological principles and of random sampling to verify reported data [27].

A recent literature review identified methodological quality assessment tools available for primary and secondary medical studies, including research pertaining to epidemiological questions such as prevalence [28]. Among the 27 tools described [28], two appeared to be relevant to the objectives of this study: the Joanna Briggs Institute’s Checklist for Prevalence Studies (from the University of Adelaide’s Faculty of Medical Sciences in South Australia) [29] and the US Agency for Healthcare Research and Quality (AHRQ) methodology checklist for prevalence study quality [30]. Both checklists proved relevant and covered established criteria for disease reporting described in the epidemiological literature. We deemed neither of the two checklists as sufficiently comprehensive to serve as a standalone tool for appraising the quality of data acquisition, reporting, and interpretation in the current context of the COVID-19 pandemic. Thus, we based our appraisal on standard criteria for rigorous conduct and reporting of epidemiological research from the established literature, including consistent and standardized case definitions, use of appropriate denominators to report case statistics, and representative samples for testing.

Two members of our research team (S.L., T.S.) reviewed the provincial and territorial websites and the news outlets. The list of COVID-19 data items they extracted appears in Table 1. We created graphs presented in this article in R statistical software [31] using the ggplot2 package [32].

Table 1 Data extraction form
Full size table

Results

COVID-19 case definitions, use of denominators, and data sources

Figure 1 displays the types of COVID-19 case definitions we found on each provincial or territorial website. Figure 2 presents the epidemiological reporting standards (case definitions, use of denominators to report COVID-19 case counts, and data sources) of the 10 provinces and for the 15 news outlets at each extraction point.

Fig. 1
figure 1

COVID-19 case definitions used across Canada. Population sizes of each province and territory have been included (as of 23 July 2021) [80]

Full size image
Fig. 2
figure 2

Epidemiological reporting standards of a Canadian Provinces and b Canadian News Outlets

Full size image

At the first extraction point (28 April 2020), only 6 of 13 provinces or territories included COVID-19 case definitions on their respective governmental websites. All provincial and territorial websites reported COVID-19 case numbers as absolute values without a denominator such as population size, per 100,000, or number of tested individuals. None reported a source of their data. By 2 June 2020, all 10 provinces and 2 of 3 territories displayed case definitions; Nunavut was the exception. British Columbia provided the source of its data, an easily accessible one. By 29 June 2020, Alberta, Saskatchewan, and Ontario joined British Columbia in providing data sources. By 15 September 2020, case definitions and sources showed no additional changes. But Newfoundland began to report certain COVID-19 case statistics with denominators.

By 15 January 2021, case definitions showed no changes, and Quebec [33] started providing sources for their data. In addition, Newfoundland [34] and Ontario [35] began reporting COVID-19 case statistics with denominators. By the final extraction point (19 August 2021), only two additional major changes occurred: Nova Scotia [36] reported COVID-19 case statistics with denominators and provided sources to their data, and Newfoundland [34] ceased use of denominators.

Among the 15 news outlets, two-thirds reported absolute case counts without applying any denominators. Few news outlets reported any data sources or provide links directly to those data. Others indicated use of “Government Sources”.

COVID-19 symptomatic versus asymptomatic testing

By 28 April 2020, all 13 provinces and territories recommended COVID-19 testing for individuals who had recently travelled out of the country and had reason to believe they had been exposed to COVID-19, or for those experiencing symptoms of COVID-19 (Table 2).

Table 2 COVID-19 symptoms or testing criteria (or both) and data reported (as of 19 August 2021)
Full size table
Table 3 Examples of news outlet reporting of COVID-19 pandemic
Full size table

COVID-19 testing criteria of all 13 provinces and territories remained unchanged at the second extraction point (2 June 2020). By 29 June 2020, the official governmental websites of Alberta, Saskatchewan, and Manitoba explicitly stated eligibility for certain asymptomatic individuals to undergo COVID-19 testing. Alberta provided the broadest testing, allowing any individual to be tested whether or not that person had any symptoms [37]. Manitoba’s COVID-19 testing guidelines allowed testing of asymptomatic individuals, or patients who visited an emergency department, or those admitted into acute care or long-term care facilities [38]. Saskatchewan offered COVID-19 asymptomatic testing only to immunocompromised individuals [39].

By 15 September 2020, most provinces and territories promoted testing of symptomatic individuals on their websites. Alberta and Saskatchewan remained the only two provinces to explicitly encourage testing of asymptomatic individuals. Saskatchewan broadened its testing capacity to allow anyone to receive COVID-19 testing. Manitoba “… developed several options for testing, including introducing voluntary asymptomatic testing for clients in a number of health care settings and for truck drivers travelling outside of Manitoba to further monitor the presence of COVID-19 in the province” [40].

By 15 January 2021, most provincial and territorial websites still predominately promoted testing of symptomatic individuals. There were, however, some caveats for testing of asymptomatic individuals. These included: individuals having had close contact with a COVID-19 positive person [41,42,43,44], individuals requested to test by public health authorities [41, 42], or people who received an exposure notification via the Canadian COVID Alert app [41, 42]. Ontario also identified certain groups as eligible for asymptomatic testing: workers of long-term care facilities, homeless shelters, or other shelters; farmers; Indigenous people; individuals requiring a COVID-19 test prior to surgery; international students who had completed a 14-day quarantine; and individuals who received a positive result from a COVID-19 antigen test [42]. Only two provinces offered asymptomatic testing to any individual: Saskatchewan [39] and Nova Scotia [44]. Previously Alberta [45] and Manitoba [40] had offered asymptomatic testing, then paused by 15 January 2021.

By 19 August 2021, most provincial and territorial websites still predominately promoted testing of symptomatic individuals with the caveats noted above. At that time, Saskatchewan [39], Quebec [41], New Brunswick [46], Nova Scotia [44], and the Northwest Territories [47] allowed asymptomatic testing.

COVID-19 case data by age, sex, and racial or ethnic minority status

Table 2 displays the types of data reported by provinces as of 19 August 2021. As of 15 July 2020, all 10 provinces reported age-stratified data on COVID-19 cases but only Quebec reported age-standardized mortality data [48]. Seven out of ten provinces (British Columbia [49], Alberta [50], Manitoba [51], Quebec [48], Ontario [52], Nova Scotia [36], and Prince Edward Island [53]) provided sex-stratified data. New Brunswick did provide sex-stratified data, but only for COVID-19 tests; it was not clear whether this represented a breakdown of positive test results or of all testing performed in general [54]. None of the territories (Northwest Territories, Yukon Territory, or Nunavut) reported any age- or sex-stratified data on COVID-19 cases. No province or territory reported any COVID-19 data stratified by race or ethnicity. By 15 September 2020, Quebec [48] no longer reported sex-stratified data. By 15 January 2021, reporting remained the same except Quebec [33] began to release sex-stratified data. As of 19 August 2021, reporting remained mostly unchanged, except Quebec [48] and Prince Edward Island [53] no longer released sex-stratified data, and New Brunswick [54] no longer released sex-stratified and age-stratified data.

Figure 2 demonstrates that by the final extraction time point (19 August 2021), fewer than half of the news outlets reported COVID-19 case statistics with a denominator and provided sources of data. (One we assessed, Huffington Post Canada, ceased operation as of 9 March 2021.)

Discussion

The purpose of this article was to identify and compare the COVID-19 case definitions of all Canadian provinces and territories, to illustrate that COVID-19 case data routinely published and disseminated to the general public is not representative of the respective target populations, and to explain why the reporting and comparing case counts without appropriate denominators may misguide public health policy. We believe it is important to appraise and understand variations in governmental and news outlet reporting of COVID-19 to the public because reporting may be unintentionally biased which may result in neglect of key public health guidelines.

Relevance of using appropriate denominators

Our assessment of COVID-19 case reporting (as of 19 August 2021) revealed that 11 of 13 provincial and territorial websites and 8 of 15 news outlets reported COVID-19 case counts only as absolute numbers. Two exceptions, Nova Scotia [36] and Ontario [35], reported case counts in reference to denominators, COVID-19 cases per 100,000 population. At the penultimate extraction point, Newfoundland [34] had reported case statistics with denominators, but by the final extraction point, no longer did so. The reporting of only absolute COVID-19 case counts prevents accurate comparison of the disease spread across geographic regions. Fixed denominators such as population size, however, have only limited utility when assessing the spread of disease over time, as eligibility criteria for testing and testing capacities vary widely, even within one region. Positivity rate, the proportion of all tests performed that are actually positive (in a given period of time) [55], is another measure used in public reporting. Low positivity rates indicate low viral prevalence and adequate surveillance capacity (amongst the tested); high positivity rates reflect high viral prevalence or testing strategies focused primarily on symptomatic individuals, or both. Despite its prevalence in news reporting, positivity rates may be biased due to differences in test-seeking or care-seeking behaviour of individuals [1], asymptomatic cases of COVID-19 [56], changes in testing capacities, and imperfect test sensitivity [57]. Obtaining accurate estimates of the burden of the disease is crucial to informing the public health response [2]. Despite this fact, news outlets and governmental bodies remain inclined to compare the disease prevalence and incidence across cities, regions, and countries (Fig. 3). For example, news outlets repeatedly called Montreal the “epicentre of the pandemic” [58, 59] in Canada, as did public health officials [60, 61] based on its high absolute number of COVID-19 cases. Even Prime Minister Justin Trudeau [62] expressed concern for Montreal residents (Table 3). None pointed out that Laval, a city north of Montreal and the third largest city in Quebec following Montreal and Quebec City, experienced similar proportions of positive COVID-19 cases and death rates of COVID-19 as Montreal (Fig. 4).

Fig. 3
figure 3

source: Government of Quebec (as of 19 August 2021): a Absolute case counts and b proportions (cases/population size). Note These curves do not reflect changes in testing capacities or selection of individuals being testing over time

Comparing COVID-19 case statistics in Montreal and Laval (Quebec, Canada). Data

Full size image

Symptom-based testing predominates

We found that most provincial and territorial websites recommended COVID-19 testing primarily to individuals experiencing symptoms of COVID-19. And some provinces (British Columbia and Ontario) explicitly discouraged testing of individuals without symptoms: “if you don’t have any symptoms, testing is not recommended even if you are a contact” [43] and “[Public Health Ontario] does not currently recommend routine testing of asymptomatic persons for COVID-19.” [63]. By 15 September 2020, only the websites of Alberta [37], Saskatchewan [39], and Manitoba [64] explicitly noted availability of COVID-19 testing to asymptomatic individuals or certain priority groups. By 15 January 2021, however, Alberta and Manitoba had paused their asymptomatic testing. By 19 August 2021, more provinces (including Quebec, Nova Scotia, and New Brunswick) allowed for asymptomatic testing.

Despite the content of postings for the public on provincial and territorial websites, in practice, COVID-19 testing may be more widely available. For instance, Public Health Ontario recommended that healthcare providers “should continue to use their discretion to make decisions on which individuals to test [for COVID-19]” [63]. Additionally, although not always stated explicitly on the websites, some provinces may have expanded COVID-19 testing for priority groups such as healthcare workers, first responders, teachers, immunocompromised individuals, patients who had been admitted to acute care hospitals, amongst others. COVID-19 case statistics that rely on symptomatic testing may grossly underestimate the true extent of spread of the epidemic. Thus, these findings highlight the need for large-scale representative testing to enable accurate estimation of the epidemic’s scale and dynamics.

Despite the importance of representative testing, it has taken nearly 1.5 years after the initial lockdowns in Canada for health authorities to apply this sampling approach in specific contexts. On 6 July 2021, Statistics Canada and the COVID-19 Immunity Task Force released preprint data on COVID-19 seroprevalence of > 10,000 Canadians tested between November 2020 and April 2021 [65]. Researchers found that between those dates, 2.6% of Canadians had COVID-19 antibodies, another 1% had the antibodies due to vaccinations. (COVID-19 vaccines were not widely available during the survey period) [65].

Lack of data on racial and ethnic minorities

Our analysis of COVID-19 reporting by governmental websites also uncovered absence of any reporting on race or ethnicity. After nearly two years since the initial lockdowns in Canada, no province or territory reported any COVID-19 data on race or ethnicity. This failure prevents Canadian public health authorities from understanding how COVID-19 impacts these groups. This is particularly problematic given growing evidence that COVID-19 disproportionately affects racial and ethnic minorities. According to the COVID Racial Data Tracker Project, a collaboration between The Atlantic and Boston University aimed at gathering race and ethnicity data on COVID-19 in the USA, “nationwide, Black people are dying at 1.5 times the rate of White people” [66]. Other racial and ethnic minority groups are also adversely affected; Indigenous and Latinos experience mortality rates of 138 and 121 deaths per 100,000 respectively compared to 98 per 100,000 for White Americans (as of 26 January 2021) [66].

These disparities may be attributed to inequities in the social determinants of health such as access to healthcare, socioeconomic conditions (including poverty and the stress that accompanies it), housing, and occupation [67]. An additional explanation that must not be ignored is systemic racism, a term used to convey “racism [that] is embedded in the policies [and practices] of public and private institutions” [68]. Systemic racism can exist even if no one in the institution is racist, but historically architects of the system and structure of the institution built these in a way that favours certain groups over others. Racial and ethnic minorities are more likely to be low-income, frontline workers (healthcare workers, caretakers, delivery drivers, amongst others), and live in housing and multi-generational homes [69, 70] under “conditions ripe for [the] spread of coronavirus” [71].

A Statistics Canada report found that neighbourhoods in Quebec, Ontario, and British Columbia with the highest proportions of visible minority residents (> 25%) had an age-standardized COVID-19 mortality rate per 100,000 population at least two times that of neighbourhoods with less than 1% visible minority residents [72]. These results align with those reported earlier by CBC News in Montreal [69] and Toronto [70]. The data gaps in race and ethnicity led to numerous calls to collect COVID-19 data on race and ethnicity from committees and community groups in Montreal [73, 74], Toronto [75], Vancouver [76], and Nova Scotia [77]. These data would allow us to better understand changes in the COVID-19 pandemic and identify the most vulnerable at-risk groups.

Limitations

This study’s limitations stem from its design. As only two members of the research team reviewed governmental and news outlet websites, we may have missed some data. We attempted to mitigate this by reviewing the sources at least twice at each extraction point and by consulting the Internet Archive (www.archive.org). Another limitation is the lack of consistent time intervals between the data extraction points. We found, however, that the data reporting methods of the governmental bodies and news outlets did not evolve as rapidly as the COVID-19 pandemic itself.

Conclusion

Accurate monitoring of the course of the COVID-19 epidemic is critical for determining which population-wide measures are necessary (or unnecessary) to prevent spread of the disease or subsequent ‘waves’ of the pandemic, or both. With the currently implemented measures on testing for, and reporting of COVID-19 cases, it is statistically difficult to arrive at a valid numerical projection of reality. As travel restrictions are still in place and household members likely being accessible to those gathering data, (repeated) random sampling of the population to be tested would enable a more accurate and precise estimate of the status and development of the epidemic. Random sampling and symptom-independent antibody testing would enable us to learn about the percentage of the population still at risk of acquiring a COVID-19 infection or who have already been exposed to the virus (seropositive) [78, 79]. This would help citizens and policy makers understand the actual scale of the ongoing epidemic and provide invaluable guidance on which preventive measures are most effective, thus necessary, and which are not, now, or soon. We also need greater focus on monitoring clinically relevant case trajectories such as people affected by COVID-19 requiring hospitalization or intensive care (rather than counts of positive tests). Understanding these case statistics in relation to available healthcare capacities is imperative as they are the key indicators of the direct impact of the ongoing pandemic on the health system and people's lives.