Developing content for national population health surveys: an example using a newly developed sedentary behaviour module
While physical (in) activity surveillance has grown and continues to grow globally, surveillance of sedentary behaviour is in its infancy. As surveillance evolves to meet the changing nature of these behaviours, there is a need for the development of national health survey questions to provide accurate and consistent measures over time. The development of national health survey content is a complex, detailed and often undocumented process. The objective of this paper is to outline the process that the Public Health Agency of Canada (PHAC) and Statistics Canada took in partnership with academic experts to develop a short, flexible, sedentary behaviour module for the Canadian Health Measures Survey (CHMS) and to provide an approach for the development of future survey content.
Development of the module followed a multi-step process. The results of this paper describe this process and present a framework for content development.
Initially, PHAC and Statistics Canada analysts worked together to identify key content required for a potential survey module. Next, this work was formalized through a contract with academic experts, the scope included a: review of existing Canadian sedentary behaviour modules; literature review linking different sedentary behaviours to health outcomes; and, international scan of modules currently in use in large national health surveys and research. The key output from both review processes was recommendations for a short sedentary behaviour questionnaire module (International Sedentary Assessment Tool). These recommendations provided an evidence-informed basis for discussions about how to revise and update the CHMS sedentary behaviour questionnaire content. Qualitative testing was undertaken and a final module was developed using survey design best practices.
Content volume in national surveys is limited due to demand to measure core content in addition to emerging health topics while keeping surveys as short as possible. Questions must therefore, be concise, valid/reliable, evidence-based, and developed using best practices. The paper describes the development process of a new survey module addressing the emerging area of sedentary behaviour for use in a national survey that may serve as a model for future population survey content development.
KeywordsSurvey Questionnaire Population health Sedentary behaviour
Canadian Community Health Survey
Canadian Health Measures Survey
International Sedentary Assessment Tool
Physical Activity, Sedentary behaviour and Sleep
Public Health Agency of Canada
Questionnaire Design Resource Centre
In 2012, the Lancet Physical Activity Series Working Group used data from the WHO global health observatory data repository to report on the global prevalence of physical inactivity . While it is recognized that physical (in) activity surveillance has grown and continues to grow globally, surveillance of sedentary behaviour is in its infancy . Relative to physical activity, the field of study around sedentary behaviour is young, and only in the past 10 years has it received a surge of attention . Sedentary behaviour is often confused with physical inactivity (i.e. not meeting physical activity guidelines), but they are not synonymous. Rather it refers to the time spent sitting, lying or reclining during waking time and includes activities such as watching television, passive transportation (e.g. riding in a bus, train or car) and using a computer at a traditional desk . Sedentary behaviour is largely recognized as an independent risk factor for chronic disease and mortality .
Canada has a long history of reporting on physical (in)activity . Data on physical inactivity and sedentary behaviour are collected using various tools on several Statistics Canada health surveys including the Canadian Health Measures Survey (CHMS), the Canadian Community Health Survey (CCHS) and the Canadian Health Survey on Children and Youth. The Public Health Agency of Canada (PHAC) includes physical (in)activity as part of its national surveillance strategy around risk factors for non-communicable diseases using data from the CHMS and CCHS. These data have been, and continue to be, used to inform policy development and strategies around healthy living, with screen time being a specific sedentary behaviour of focus in the Canadian 24-Hour Movement Guidelines for the Early Years  and Children and Youth , ParticipACTION’s Report Card , and Canada’s Common Vision for Increasing Physical Activity and Reducing Sedentary Living . These combined initiatives have helped to place Canada at the forefront of leaders in the field of sedentary behaviour. Canada continues to push boundaries in this area and is working towards improving its surveillance data on sedentary behaviour.
Alongside the surge in sedentary behaviour research, there was growing recognition within the PHAC and Statistics Canada, from both a policy and surveillance perspective, that there was a need to move beyond simply describing physical inactivity as it related to Canadians who did not meet physical activity guidelines. Rather, concepts such as sedentary time (including screen time), light intensity physical activity, and sleep, alongside their determinants and positive and negative health benefits, were important behaviours to include. It was essential to move beyond measuring adherence to physical activity guidelines [6, 7, 10] alone, and begin to examine and understand other intensities of movement during the rest of the day. From a surveillance perspective, it was important to understand how we might define these behaviours and their key components (e.g. screen time, reading time, non-active travel) accurately and measure them consistently over time at the population level. The growing public health need to monitor and report on these behaviours and related factors led to a modernization of the PHAC’s physical activity surveillance system which is now directed by the Physical Activity, Sedentary behaviour and Sleep (PASS) Indicators .
With the development of the PASS Indicators, it was clear that the PHAC had many ready-made indicators for physical activity, but that sedentary behaviour and sleep had fewer existing indicators to draw upon. Even with the long history of measuring screen time within Canadian national health surveys, it was clear from consultations between the PHAC and Statistics Canada that there was a need to update indicators and their associated measures to keep up with the sedentary practices of Canadians (especially around screen time consumption) and to consider sources of sedentary time beyond screen usage (e.g., commuting, sitting at work etc.) which were emerging in the scientific literature. For example, the original measures of screen time on earlier cycles of the CCHS did not include advances such as tablets and the use of social media on phones . While efforts had been made over time to keep up with these societal changes in the CCHS and CHMS questions, ultimately it became clear that a more focused approach based on a thorough examination of the scientific literature and international survey best practices was warranted.
While survey methodology is often part of formal education, applying these methods in ‘real-world’ settings can be challenging. Further, the development process of national survey content, as well as how it is established is rarely described. We felt that presenting the methods undertaken in the development of this module would be of interest to those learning and applying these methods in other countries, contexts or towards other issues. In the case of surveys, content and questions must be based on the best available evidence, be valid and reliable, be relevant, have a high enough prevalence to be measurable with the survey’s sample size, and address important priorities. In the case of Canadian national surveys, they also need to be concise due to limited space and time and understandable for the general public to ensure quality responses. Understanding these factors is important for meeting the needs of national surveillance systems. The objective of this paper is to outline the process that the PHAC and Statistics Canada took in partnership with academic experts to develop a short, flexible, sedentary behaviour module for use on a national population survey (the CHMS) in order to improve the accuracy and reporting of sedentary behaviour in Canada. By outlining the development process of the sedentary behaviour module, the authors hope to provide important context and a framework for future module development and for those interested in using the module.
Understanding the evidence base
With the content of the 2020 cycle of the CHMS up for review, an opportunity to examine and revise sedentary behaviour content presented itself. To begin with, a contract was established between the PHAC and a leading expert in the field of sedentary behaviour (TJS) to provide a formal report on: 1) the evidence around specific types of sedentary behaviour (e.g. total sedentary time, screen time, reading, passive travel, etc.) related to health outcomes in order to provide context for question development and prioritization; and, 2) suggestions for a short sedentary behaviour module focusing on the most reliable and valid questions available from both the published literature, as well as national health surveys from around the world. Further instructions included that the module be developed to provide questions in order of prioritization with respect to their associated risks via the evidence-base. For example, if only one question could be posed, what would this look like or if only two questions could be posed, what would they include, etc.
The expert was free to involve or consult other experts as they thought appropriate and were encouraged to publish their findings. The contract resulted in a report back to the PHAC which included a proposed module, and subsequently, an article in a peer-reviewed, open-access journal describing the state of the evidence around the reliability and validity of self-report questions/questionnaires used in both research and national health surveys . The expert report proposed a new sedentary behaviour module, entitled the International Sedentary Assessment Tool (ISAT) available for download here: http://www.sedentarybehaviour.org/sedentary-behaviour-questionnaires/. The module consisted of six sedentary questions supported by the literature; based on the most valid and reliable tools available. The questions were listed in order of prioritization with respect to their known association with health outcomes and provided an option for a single or two-item screen time question (depending on room on the survey). Finally the questions drew on current trends in sedentary behaviour including the importance of multi-tasking (e.g. sitting while watching TV and simultaneously using a smart phone) and changing types of screen media (e.g. tablets, smart phones, e-readers).
Module development process
Sample of suggested survey design best practices and their application in the sedentary behaviour module development
Survey design best practice concept
Application to sedentary behaviour module
Identify all of the concepts to be measured
Identified and defined sedentary behaviour and its types and domains.
Modify writing of the questions to read at a grade 10 level
Adjusted previously validated questions to read at the grade 10 level by using appropriate language.
Cover only one topic/concept per question
Each question posed focuses on one type/domain of sedentary behaviour (e.g., sedentary time, screen time, travel, reading).
Break complex or conditional concepts into multiple questions
Recognizing that sedentary behaviour is engaged in differently during the week and weekend, each question includes a ‘per weekday’ and ‘per weekend day’ designation.
Phrase questions positively and avoid negative structures (i.e. don’t say: How often do you not go out because you cannot afford it? Better phrasing would be: How often do your finances affect your ability to go out?)
No negative structures are used in asking about time spent sedentary.
Avoid the use of leading or biased questions
There are no references to the sedentary behaviour of others in the questions.
Create a logical order and flow to ensure the process feels like an interview rather than an interrogation
We begin by asking respondents to estimate total sedentary time, and then ask about specific sedentary behaviours.
Design sensitive questions to ensure that their relevance is obvious or at least easy to explain
While none of the questions are ‘sensitive’, considerable thought was given to provide examples of pertinent activities under each question.
Keep the questionnaire as short as possible to ensure high response rates and to avoid partially completed interviews
The module was designed to be modular and as brief as possible. Each question can be asked on its own and questions can be added or dropped in response to the final estimate of available time for the survey.
Try to use consistent scales to avoid confusing respondents
Response options for all questions were standardized to reflect common formatting within the CHMS i.e. hours and minutes.
Once the module was thoroughly reviewed by the PHAC, it moved back to Statistics Canada for qualitative testing. The objectives of qualitative testing included: testing respondents’ understanding of specific concepts, terminology, questions and response categories; obtaining feedback from participants regarding their overall impression of and reactions to the questions; assessing respondents’ ability and willingness to respond to survey questions; determining the appropriateness and completeness of the response categories; testing the respondent-friendliness of the questions (i.e. they are easily understood and accurately completed); and, estimating how much time the module takes to administer.
Qualitative testing was undertaken by the Questionnaire Design Resource Centre (QDRC) at Statistics Canada and included 21 one-on-one in-depth interviews; 11 English and 10 French. One observer from the questionnaire development team was physically present at each interview while other project team members observed the interviews remotely using Cisco WebEx (online meeting/video conferencing software) and provided real-time questions to the moderator. Interviews lasted no more than one hour and respondents received a monetary compensation at the end. Participants were recruited based on a variety of factors and identified: youth aged 12–17; parents of children aged 3–11 years (proxy for child responses as per CHMS methodology); people who had broken or fractured bones; people with arthritis or osteoporosis; and, those who spent time in the sun in the past two months. The last three categories were chosen to accommodate other modules that were simultaneously tested.
Results of the qualitative testing were summarized in a report from the QDRC and distributed to the project team. The report identified areas of concern via the respondents for the three tested questionnaire modules. With respect to the total sitting time question, qualitative testing revealed that respondents often excluded sitting time that occurred in domains outside of leisure time (e.g. travel, work/school, household) and therefore, total sitting time was likely under-reported. However, it was noted that placing the total sitting time question following the sub-domain questions (screen, reading, transport) led to respondents recalling only sitting time that occurred outside of the other domains. Respondents largely commented that the week and weekend day delineation was important given the changing nature of their behaviours based on the type of day; however, participants who were retired commented that sitting time was similar regardless of week/weekend status. Many respondents identified that it was not always clear that they were being asked to report daily totals versus weekly totals. Most respondents found it easier to be asked a single screen time question versus two which separated out TV from computer-type screen time. Parents who proxy-reported sitting time for their children reported it was difficult to report time spent sitting during school hours.
This paper describes the rigorous process that the PHAC and Statistics Canada undertook to develop a new sedentary behaviour module for the CHMS (and future population-based surveys). The number of questions in the module is limited due to time constraints given that the CHMS collects information on many factors related to the health and health habits of Canadians (i.e. there is a limited amount of time allotted for asking questions to reduce respondent burden and maintain response rates). Due to these time constraints, other types and domains of sedentary behaviour such as workplace sedentary time were not included in the final module; the evolution of evidence will likely bring suggestions for future revisions and it is important that these changes (e.g., workplace sedentary time) be considered using a similar development process.
We recognize that validation of the module is important once data is collected. It is essential that as new modules are developed and used in surveys that their reliability and validity are examined to evaluate their performance and that they are refined accordingly. The CHMS collects objectively measured movement data using accelerometers and thus offers the opportunity for future comparisons between self-reported sedentary behaviours (i.e., screen time, sitting time) and accelerometer-measured sedentary time. It is important to stress that while comparisons are possible, validation of this module’s sedentary behaviour questions is not. Accelerometers capture total sedentary time but lack any information about the context of the sedentary behaviours. In contrast, this questionnaire estimates time spent in specific sedentary pursuits (e.g., screen time) but is limited in its ability to capture every minute of sedentary behaviour incurred throughout the day. This incongruence between two measures of the same construct is a limitation and challenge of measuring health behaviours. We will assess the criterion or predictive validity, examining how these measures associate with commonly known health outcomes (e.g., cardiovascular disease, diabetes). With continued use, we will also be able to assess the reliability of the measure.
Currently many developed countries are seeking solutions to the phenomena of declining response rates on national surveys . One solution has been to shorten surveys to render them less time consuming; inevitably resulting in a greater demand for survey space. The process that created the ISAT presents a practical solution to this epidemiological phenomenon, which like many challenges, also brings an opportunity. In this case it required a greater emphasis on: thinking about the module in the preplanning phase of the survey; building in flexibility and ranked priorities into the module; and, basing the development and process on the best science and international best practices. For Statistics Canada this collaborative effort reflected their increased attention towards three key pillars of their modernization strategy including: user-centric delivery; leading-edge methods and sharing; and, collaboration. For the PHAC, the collaboration produced a documented process on the development of the sedentary behaviour module used in the CHMS and a framework for future module development (Fig. 1). This paper’s description of how the ISAT was subsequently modified for inclusion on a national health survey will hopefully provide other potential users with the information required to incorporate this module into their own survey instruments, as well as a process they might use for their own future module development.
SAP and GPB led the writing of the manuscript. KCR, PL, AMM, RCC, MF, TJS and WT contributed to the writing of the manuscript. All authors read and approved the final manuscript.
SAP is funded by a Canadian Institutes of Health Research – Public Health Agency of Canada Health System Impact Fellowship. The development of the International Sedentary Assessment Tool received financial support from the Public Health Agency of Canada.
Ethics approval and consent to participate
Consent for publication
SAP has received an equipment competition award from PAL Technologies Inc. TJS has received additional research support from Stepscount Inc., and in-kind support from Stepscount Inc., Ergotron and Fitabase. TJS has also received honoraria for presentations on the health impact of sedentary behaviour. All other authors declare that they have no competing interests related to the work.
- 2.LeBlanc AG, Gunnell KE, Prince SA, Saunders TJ, Barnes JD, Chaput JP. The ubiquity of the screen: an overview of the risks and benefits of screen time in our modern world. Translat J Am Coll Sport Med. 2017;2(17):104–13.Google Scholar
- 6.Canadian Society for Exercise Physiology. Canadian 24-Hour Movement Guidelines for the Early Years. 2018. Accessed 7 Aug 2018. Available at: https://www.participaction.com/sites/default/files/downloads/PAR7972_24Hour_Guidelines_EY_En.pdf.
- 7.Canadian Society for Exercise Physiology. Canadian 24-Hour Movement Guidelines for Children and Youth. 2016. Accessed 16 July 2018. Available at: http://www.csep.ca/view.asp?x=696.
- 8.ParticipACTION. 2018 ParticipACTION Report Card on Physical Activity for Children and Youth. Accessed 16 July 2018. Available at: https://www.participaction.com/en-ca/thought-leadership/report-card/2018.
- 9.Public Health Agency of Canada. A Common Vision for Increasing Physical Activity and Reducing Sedentary Living in Canada. 2018. Accessed 16 Sept 2019. Available at: https://www.canada.ca/en/public-health/services/publications/healthy-living/lets-get-moving.html.
- 10.Canadian Society for Exercise Physiology. Canadian Physical Activity Guidelines for Adults. 2011. Accessed 9 April 2018. Available at: http://csepguidelines.ca/wp-content/uploads/2018/03/CSEP_PAGuidelines_adults_en.pdf.
- 12.Statistics Canada. Canadian Community Health Survey – Detailed information for 2000–2001 (Cycle 1.1). 2007. Accessed 9 April 2018. Available at: http://www23.statcan.gc.ca/imdb/p2SV.pl?Function=getSurvey&Id=3359.
- 14.Swain L. Basic principles of questionnaire design. Surv Methodol. 1985;11(2):161–70.Google Scholar
- 15.The Economist Editorial Team. “Plunging response rates to household surveys worry policymakers”, The Economist, International Print Edition. May 24th, 2018.Google Scholar
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.