Correspondence/findings

The National Health Interview Survey (NHIS) is an annual in-person household survey of the health of the U.S. civilian, non-institutionalized population conducted by the National Center for Health Statistics (NCHS), Centers for Disease Control and Prevention (CDC). In 2002, 2007 and 2012 supplemental questions on complementary medicine were added to the NHIS. These questionnaires filled a void for large-scale, nationally representative datasets on the out-of-pocket costs, prevalence, and reasons for use of complementary therapies in the U.S. All three surveys were sponsored primarily by the National Center for Complementary and Alternative Medicine (NCCAM), part of the National Institutes of Health (NIH). The first NHIS complementary medicine survey, fielded in 2002, was initiated in-part by the NCCAM 5-Year Strategic Plan 2001–2005, which included the creation of a Special Populations Program to, among other goals, support research on complementary medicine use in racial and ethnic minority populations [1]. In the early 2000’s, existing datasets for complementary medicine use in the U.S. lacked sufficient sample sizes to analyze use by minority populations. Given this deficit, NCCAM made the decision to fund a large national survey of complementary therapies. The NHIS was chosen as the vehicle for the complementary medicine surveys because of its large sample size and its oversampling of non-Hispanic black and Hispanic persons (and Asian persons beginning in 2006).

Analyses of the 2002 and 2007 surveys have resulted in more than 100 publications in peer-reviewed journals [2]. It is expected that the 2012 supplement will have the same significant impact on research and policy-making communities as did the 2002 and 2007 versions, which provided the most comprehensive nationally representative data on complementary medicine use available for the United States. Although many research priorities and specific content identified during the development of the 2002 questionnaire remained as part of the 2007 and 2012 surveys, each questionnaire contained unique topics. For example, the 2012 survey included a series of items related to using complementary therapies for wellness-related reasons. NCCAM decided to include this topic during development of its third strategic plan [3], which placed an emphasis on use of complementary therapies to promote health, wellness and well-being.

This article describes the multi-faceted and largely qualitative approach taken to develop the three questionnaires, particularly detailing the results of expert input and testing of cognitive salience and clarity of questionnaire items. This article will guide researchers and policy makers who plan to use data generated from the surveys, as well as inform quantitative assessments of questionnaire validity and of the research value provided by the resulting datasets.

Overall development process

The development of the three NHIS complementary medicine questionnaires was not performed within the framework of a planned research study. Instead, questionnaire design and development generally followed steps documented in the literature [4–8] and in an NCHS report on questionnaire design [9]. The NCHS protocol included four phases: 1) planning, 2) exploratory studies, 3) developmental methods, and 4) testing methods. Planning is particularly important for government surveys which generally have tight timeframes, require multiple clearances, and may involve staff from multiple organizations. For each questionnaire, development began 18–24 months prior to data collection in collaboration with survey methodologists from NCHS. The second phase, exploratory studies, is useful for identifying the response process and response problems unique to a given set of questions. This step applies to specific aspects of complementary medicine modalities such as the relationship between self-identity and use of nonvitamin, nonmineral dietary supplements (NVNMDS) [10] discussed later in this article. The third phase, developmental methods, involves iterative rounds of questionnaire evaluation through participant interviews as the main approach for discovering problems with questionnaires. Although the number of respondents need not be large (20–50 respondents), ideally they should differ widely in demographic characteristics to maximize generalizability. The final phase described in the NCHS report, testing methods, refers to large-scale field testing. Although budget constraints did not allow for large-scale field pre-testing, input from field staff was solicited prior to fielding and resultant modifications were made.

Development of the three NHIS questionnaires applied the definition of complementary medicine established by NCCAM: “a group of diverse medical and health care systems, practices, and products not generally considered part of conventional medicine” [1]. Although the specific therapies included varied slightly across survey years, the following were included in all three questionnaires: acupuncture, Ayurveda, biofeedback, chelation therapy, chiropractic care, energy healing therapy, hypnosis, massage, naturopathy, NVNMDS, homeopathic treatment, diet-based therapies, yoga, tai chi, qi gong, and meditation and other relaxation techniques. The 2002 questionnaire collected complementary medicine use for adults only, whereas the 2007 and 2012 included children and adults. All three surveys were approved by the NCHS Research Ethics Review Board.

Specific steps detailed in this article include: (a) literature reviews, (b) feedback from scientific and practitioner communities, (c) expert input and workshops, and (d) participant interviews (cognitive testing). Additionally, for the 2012 child questionnaire, quantitative analysis of data from the 2007 child complementary medicine questionnaire (frequencies analysis of prevalence of therapies by age, race/ethnicity, disease status, and use for health conditions) and focus groups with health professionals and the lay public were conducted. Although these steps took place sequentially within the overall development process (literature review, then expert panels, then draft initial questions, then cognitive testing, then refinement), the synthesis of these components to yield the final content was an iterative process. For example, after the expert panels, during the drafting of initial questionnaire items, we often discussed a particular series of questions with the expert panel member who focused the corresponding discussion during the meeting. Likewise, while refining the questionnaire based on results of cognitive testing, we often referred back to notes from workshops, consulted with expert panel members, or sought additional research articles to decide on the most appropriate solution. Specific steps used in the development of each questionnaire varied (Table 1) and are described in discrete terms below.

Table 1 Iterative development process for complementary medicine questionnaires
Full size table

Literature reviews

The goal of each literature review was to identify previous surveys on complementary medicine use in order to help determine which therapies to include in the NHIS questionnaire. These surveys were reviewed by the expert panels when making recommendations to NCCAM and NCHS staff on what complementary therapies should be included in the NHIS. Literature reviews for all three surveys employed the search paradigm utilized by Wootton and Sparber [11]. Development of the 2002 questionnaire began by reviewing smaller-scale national and regional surveys identified by Wooton and Sparber through January 5, 2001. Development for the subsequent surveys examined U.S. national or regional surveys on complementary medicine that were published after January 5, 2001. Only U.S. surveys were considered during the literature review since the terminology, definitions and regulations vary considerably from country to country. For instance, although in the U.S. dietary supplements are regulated as food-based in the Dietary Supplement Health and Education Act [12], in other countries, such as Germany, they are often regulated as drugs [13] or, as in Japan, covered by national health insurance [14].

2002

The systematic review by Wooton and Sparber [11] identified seven population-based surveys published in peer-reviewed journals through January 5, 2001. Of these, five reported data from national population-based surveys on complementary medicine use in the U.S. [15–19] and two reported data from state-level population-based surveys – Florida [20] and South Carolina [21]. The report by Druss and Rosenheck [19] was based on data from the 1996 Medical Expenditures Panel Survey supported by the U.S. Agency for Healthcare Research and Quality. These seven surveys served as a starting point by the expert panel for identifying content to be added to the 2002 NHIS. Item wording from these surveys was often changed to be consistent with the NHIS format.

2007

In preparation for the 2007 NHIS complementary medicine survey an updated search of the scientific literature published up to January 2006 was conducted, and five additional national population-based surveys in adults [22–26] and one state-level - Michigan [27] were identified. Two of these were initial reports from the 1999 and 2002 NHIS surveys [22, 24]. When reviewing the four non-NHIS surveys, particular attention was paid to complementary approaches not included in the NHIS surveys, as well as to reasons individuals said they used complementary therapies. Among the complementary therapies included on these surveys but not on the 2002 NHIS were traditional healers and movement therapies, both of which were added to the 2007 survey.

2012

From published reports through January 2011, the search update for the 2012 NHIS identified three new national-population based surveys [28–30], three state-level population-based surveys – California [31], Hawaii [32], and Minnesota [33] - and one survey of active duty military personnel in the U.S. Navy and marine corps [34]. These were reviewed for complementary approaches not contained in the 2007 NHIS, as well as wellness-related reasons individuals used these therapies. No previously unidentified complementary therapies were found in these surveys.

In addition, for the 2012 questionnaire, literature searches were employed specifically to inform questionnaire design related to reasons and motivations for using complementary therapies, such as for wellness or well-being, as well as questions asking about insurance coverage [35–40]. PubMed and Google Scholar were searched, and additional references were suggested by outside experts attending a workshop on wellness (Section Wellness workshop) and members of a think tank (Section Research topic think tank) to identify research priorities for the 2012 questionnaire. Wellness-related reasons identified in the literature for why individuals use complementary therapies include physical, spiritual, and emotional well-being, nutrition and lifestyle, and to avoid pharmaceuticals [35–40].

To identify additional dietary supplements for inclusion in the 2012 NHIS, dietary supplement sales data were reviewed [41]. Among the many products added to the NHIS survey were Acai, bee pollen and other bee products, digestive enzymes and methylsulfonylmethane (MSM). In total, 119 nonvitamin, nonmineral dietary supplements were queried in the 2012 NHIS versus 44 in the 2007 NHIS.

Feedback from scientific and practitioner communities

Both the 2002 and 2007 surveys received considerable attention by the public, clinical and research communities. Staff at NCCAM received unsolicited feedback from complementary medicine researchers and practitioner organizations that was used to inform the development of the 2007 and 2012 surveys. Much of this feedback came during meetings held for other purposes and was not collected systematically. Because this feedback was unsolicited and non-systematic, data was not kept on who provided specific feedback and when.

2007

Based on unsolicited feedback on the 2002 questionnaire and updated literature reviews, the following changes were made for the 2007 survey: 1) Addition of questions on complementary medicine use by children and the condition being treated by these therapies; 2) addition of out-of-pocket expenditures on complementary therapies; 3) addition of movement therapies (Feldenkrais, Alexander technique, Pilates, Trager psychophysical integration); 4) addition of traditional healers to better capture complementary medicine use in Hispanic and Native American populations; 5) deletion of the megavitamin section; 6) addition of questions to find out why some adults did not use five types of complementary therapies - acupuncture, chiropractic or osteopathic manipulation, NVNMDS, yoga, and meditation - chosen based on the most commonly used therapies from each of the major categories of complementary medicine as defined by NCCAM at the time of questionnaire development [1]; 7) deletion of folk medicine; and 8) addition of questions about the relationship between complementary and conventional treatments (adult version only). Content for 2002, 2007, and 2012 questionnaires are shown in Table 2. Other changes, such as the addition of questions on vitamins and minerals, and an expanded section on NVNMDS, grew out of consultations with other parts of the NIH.

Table 2 Content of 2002, 2007, and 2012 NHIS complementary medicine questionnaires
Full size table

2012

Based on unsolicited community feedback about the 2007 survey several items were modified for the 2012 version. For example, follow-up questions on yoga were restricted to only those using meditation and/or breathing exercises as a component of yoga. Additionally, the list of NVNMDS was expanded beyond that of 2007.

Expert panels and workshops

Expert panels were convened for the development of the 2002 and 2012 surveys, but not for the 2007 survey. Panel members were invited based on their published work and expertise, and were considered experts in their own field. No sampling method was utilized. For each of the three in-person panels described below an open-dialogue process was used, with iterative summaries of high level input and priorities set throughout the meetings. The 2002 panels consisted of half-day phone calls and numerous email exchanges, while the 2012 panels were day-long meetings. Each meeting began with introductions, background presentations and an explanation of goals and objectives for the session. Input was solicited from each member of the panel and careful attention was given to keep discussions focused and on-target. Time was allocated at the end of each meeting to reach consensus on conclusions discussed throughout the meeting. For the in-person 2012 panels, scientific notetakers were utilized to ensure thorough documentation.

2002

In February 2001 a panel was convened made up of 13 national experts in survey design and complementary medicine research and practice, along with NCCAM and NCHS staff, to prioritize content for the 2002 questionnaire. A second panel made up of 13 experts in women’s health, mental health in minority populations, complementary medicine use by African Americans and Native Americans, use of Chinese medicine, and cultural relevance was assembled to review draft questionnaire items. Table 3 shows a complete listing of expertise on each panel. The second panel did not convene as a group, but expert opinion was solicited in writing from each individual member related to the person’s area of expertise and additional modifications were made based on this input.

Table 3 Expertise of panel members
Full size table

The first expert panel suggested specific therapies to include in the 2002 NHIS. The expert panel also identified four areas of research priority: 1) Prevalence of complementary medicine use; 2) purpose of complementary medicine use (whether for a specific condition, general wellness, or both); 3) disclosure of complementary medicine use to conventional providers; and 4) inventory of natural supplements and prescription medications (not fielded due to logistical reasons). The highest priority was to identify the specific conditions or diseases individuals report using complementary therapies to treat, which is distinct from assessing the prevalence of complementary medicine use among individuals who experience one or more health conditions. Analysis of this later construct is possible by linking the NHIS complementary medicine questionnaires to other NHIS core questionnaire data files. Additional conditions and diseases hypothesized to correlate with complementary medicine use were added to those already in the NHIS core questionnaire. The second panel identified ways to improve the cultural sensitivity of the questionnaire. For example, the use of the word “traditional” to refer to Western medical practices was replaced with “conventional” based on the recommendation from a cultural expert concerned that “traditional” has different meanings to individuals with regard to various culturally related practices.

Based on review of the literature and input from the panels of outside experts, a questionnaire was constructed, edited and refined by NCHS and NCCAM staff. Underlying this process was a set of criteria for questionnaire development identified from the literature [42–44] (Table 4). Once the questionnaire was constructed, the expert panels were given an opportunity to provide additional feedback. The draft questionnaire was also circulated to experts throughout the NIH for comment.

Table 4 Criteria used to develop items for complementary medicine questionnaires 1
Full size table

2012

Wellness workshop

As evidenced in NCCAM’s strategic plans, the Center’s priorities have evolved during the past decade from a focus on treatment of disease to symptom management and promotion of optimal health [1, 3]. In August 2009, NCCAM staff sponsored a one-day workshop on how to best capture the concept of wellness and optimal health via survey items. Twenty-seven individuals participated in the workshop including behavioral scientists from universities and non-profit organizations, as well as representatives from NCCAM and other parts of NIH, CDC, and the Food and Drug Administration (Table 3). Because it was the sole internationally accepted definition at the time, NCCAM utilized the World Health Organization’s definition of wellness to generate discussion: “wellness is the optimal state of health of individuals and groups. There are two focal concerns: the realization of the fullest potential of an individual physically, psychologically, socially, spiritually, and economically, and the fulfillment of one’s role expectations in the family, community, place of worship, workplace and other settings” [45]. In preparation for the workshop, the planning committee identified criteria to help guide discussion [46–56].

Workshop attendees discussed several topics including how best to define wellness and differences between wellness and well-being. Discussion at the workshop focused on psychological, physical, spiritual, and social measures of wellness, as well as the relationship between wellness and use of complementary therapies. Workshop results guided the development of wellness items for the 2012 survey, particularly the interviewer protocol for cognitive testing of the wellness concept.

Interviewer protocols are guides to assist in the conduct of qualitative interviews and generally contain verbal probing and follow-up questions to aid in providing some standardization across interviews [57]. The interviewer protocol used for the testing of the wellness concept contained questions to elicit reasons or motivations for using complementary therapies without presupposing a “treatment” reason and respondents were asked about changes noted in “mind or body” when using the therapy (see section Concepts of wellness and well-being).

Research topic think tank

In September 2010 NCCAM staff convened a panel of experts to review the previous two NHIS complementary medicine questionnaires and identify items to retain as well as key research areas not previously covered. The panel was comprised of 24 individuals with expertise in health services research, minority health, economics, and psychometrics, as well as complementary therapies (Table 3). Based on feedback from the research topic think tank, an extensive set of questions was included in the draft questionnaire about insurance coverage and out-of-pocket expenses. This draft set of questions captured detailed information about deductible amounts, number of visits and other dollar amounts covered by insurance, and whether the respondent’s complementary medicine use was reduced due to limited insurance coverage. This section was ultimately pared down due to complexity and time restraints (see Section Draft questionnaire).

Members of the think tank also recommended that deep breathing be removed as an independent intervention in the survey given the wide variability in how participants interpret the question as identified during cognitive interviews. Instead, queries on the use of deep breathing exercises were imbedded as follow-up questions for participants using yoga, tai chi, qi gong, meditative techniques, hypnosis and biofeedback. Finally, the think tank recommended that the survey provide some specificity on the types of meditation being used. Thus, instead of asking a global question as to whether a participant used meditation, the 2012 survey asked about three broad classes of meditation identified in an Agency for Healthcare Research and Quality systematic review: mantra, mindfulness and spiritual [58].

NCCAM staff members reviewed successive drafts for face validity and item phrasing, and an NCCAM staff member trained in cognitive interviewing techniques administered the draft questionnaire to nine NCCAM staff members with various cultural and academic backgrounds to better refine the questionnaire before it underwent cognitive testing with volunteers from the general public.

Cognitive interviewing

Cognitive interviewing, a technique to learn respondents’ thought processes as they answer survey questions, has become standard practice for survey development [59]. The aim is to uncover how respondents interpret the intent and meaning of survey questions, and whether or not these match those of the researcher. The interview generally involves one interviewer and one respondent, usually takes between 30 to 90 minutes, and respondents are often compensated for their time. The interviewer will read the draft questions to the respondent and then use probing techniques to determine how the respondent is interpreting the questions and what the respondent is thinking while answering. The interviews were videotaped and their verbal content entered into Q-notes, a software system created by NCHS’s Questionnaire Design Research Laboratory (QDRL) staff to organize and help with the analysis of cognitive interview data. Interpretations of key terms and concepts in each question were examined and compared among participants in order to identify inconsistent interpretations. Ideally, interviewing continues until “saturation” is reached, that is, no new data are emerging and all the concepts are well defined and explained [60]. However, because the NHIS must begin data collection in January each year, real-world deadlines did not allow for saturation for all sections of the questionnaire; rather, cognitive interviewing focused mainly on new topic areas added in each survey year, or existing sections that were substantially expanded (e.g. NVNMDS in 2007).

After receiving IRB approval, QDRL staff used a variety of recruitment methods to obtain volunteers for cognitive testing including advertisements in the Washington Post and alternative medicine magazines, flyers placed at various co-ops and yoga studios, and the offices of various providers of complementary therapies. Informed consent was obtained from all participants prior to the start of the interview. For each survey year, a total of 25–48 in-depth semi-structured or open-ended interviews were conducted with volunteers from the general public with a range of socio-demographic characteristics (Table 5). During these interviews, questionnaire concepts were tested for cognitive salience and clarity. Possible actions taken as a result of cognitive testing included accepting the original question, accepting the original question with minor edits, accepting the original question with major edits, dropping the original question, and writing a new question (Table 6). The questionnaire development process was fluid and iterative so that changes were made and the questionnaire re-tested throughout the development process. Table 7 provides additional examples of problems found during cognitive review of all three survey questionnaires and the resulting modification made in order to reduce response error.

Table 5 Demographic characteristics of participants in cognitive interviewing, 2002, 2007, 2012
Full size table
Table 6 Examples of actions taken based on results of cognitive review and focus groups, 2012
Full size table
Table 7 Examples of response error in cognitively tested questions for 2002, 2007, and 2012 questionnaires
Full size table

2002

Twenty-five in-depth semi-structured interviews were conducted; except for chelation therapy, collectively, interviewees had used every complementary medicine identified by literature review and outside experts for inclusion in the questionnaire. Two main findings quickly emerged: 1) modalities were practiced differently with respect to seeing a practitioner or using on one’s own; and 2) many of the initial modalities tested were not discrete and overlapped with other modalities tested. To mitigate the first problem, NCHS staff, with input from the expert panel, identified 10 therapies that are mainly practitioner-based. For these therapies, the scope was narrowed to ask “Have you ever seen a provider for…” rather than “Have you ever used…” This better matched the way respondents described using complementary therapies during interviews. To reduce overlap among therapies, “Chinese medicine” was dropped because it was found to encompass many of the more specific therapies such as acupuncture and NVNMDS. In subsequent versions of the questionnaire, “home remedies” was tested as a self-care therapy and was deleted because it was found to mean different things to different respondents, as well as overlap with NVNMDS, vitamins, and homeopathy. Ultimately, a wide range of modifications and restructuring took place based on the results of cognitive testing including: 1) practitioner-based modalities were separated from self-use modalities; 2) similar modalities were consolidated (e.g. yoga, tai chi, and qi gong were combined) or deleted (e.g. Chinese medicine); 3) questions that lacked meaningful responses or consistent interpretations were deleted (e.g. age of first use, “bad reactions,” and “home remedies.”) [61].

2007

Thirty-two cognitive interviews were conducted on seven different versions of the 2007 draft questionnaire. Results from testing reaffirmed two issues that surfaced during cognitive testing of the 2002 questionnaire. The first relates to home remedies, which was re-tested for the 2007 questionnaire as a practitioner-based therapy, with the goal of capturing visits to complementary medicine practitioners in minority and ethnic populations. However, testing revealed that respondents often do not consider home remedies to be practitioner-based, even when the question text includes the phrase “Have you ever seen a practitioner for…” Although respondents spoke of both home remedies and folk medicine as practices passed down through the family, as found during cognitive interviewing for the 2002 questionnaire, the two terms meant different things to different participants and were therefore deleted. Second, heavy users of complementary therapies had difficulty recalling how old they were when they first used a particular therapy even when provided age categories. Because testing showed that estimates would be less reliable for long-time users of complementary therapies compared to recent users, the question was deleted [62].

In addition to these findings, cognitive testing also revealed that the question on “deep breathing exercises” produced numerous false positive responses. Many respondents had very little knowledge of this technique and essentially guessed at its meaning. One respondent said yes to having used deep breathing because “this is when you let out a sigh of being home to get your second wind to cook dinner.” [62]. To try to mitigate this problem, a definition was added to the term, but it was ultimately dropped from the 2012 survey based on Think Tank recommendations (Section Research topic think tank). Particular attention was also given to three sections that were new and/or expanded from the 2002 questionnaire (see below).

Child section

For the 2007 NHIS, NCCAM expanded the questionnaire to include a limited set of items on complementary medicine use by children, a topic that had not been previously studied on a national level. The NHIS child section is answered by proxy, usually the child’s parent. As expected, testing revealed that parents do not always know the answers to items on motivations and behaviors related to their children, such as whether a child prays for his or her health. Questions on motivations and prayer were, therefore, not asked in the child section.

Mineral and NVNMDS sections

Based on interest and funding by the NIH Office of Dietary Supplements, the 2007 questionnaire included expanded sections on vitamin and mineral supplements and NVNMDS. These sections collected detailed follow-up information for up to two specific vitamins or minerals or NVNMDS supplements and an expanded list of reasons for using these. Although the sections on vitamins and minerals and NVNMDS were identical, cognitive testing revealed problems unique to the NVNMDS items [10, 62]. First, it became apparent that respondents lacked a consistent, agreed-on definition of an herbal supplement. Respondents held wide-ranging and varying definitions of NVNMDS and efforts to craft a definition that met researchers’ criteria and was understandable to respondents fell short. Ultimately, the best approach was to simply ask respondents about taking specific supplements listed on a flashcard. This made the survey response task concrete and less reliant on shared definitions, although, the tradeoff was the loss of information about supplements not listed on the card. Second, some respondents viewed themselves as having or not having a “complementary medicine identity” and this perception affected the way they interpreted and answered questions, such that response error occurred even when the term “natural herbs” was understood as intended. To mitigate this, the description of “a typical herb user’s” lifestyle was removed from the lead-in statement [10, 62].

Perceptions of complementary therapies

A section on perceptions of complementary therapies was explored for the 2007 questionnaire. The questions were to capture respondents’ perceptions about the scientific evidence, safety, effectiveness, and holistic aspects of complementary therapies, as well as items concerning locus of control and how decisions are made to use or not use complementary therapies. The initial plan was to ask a long series of items about each individual complementary therapy, but time constraints made this unfeasible. QDRL staff then tested a version asking about complementary therapies as a whole, and determined that respondents’ views vary by specific therapies such that they could not arrive at a single answer. Ultimately the section was drastically reduced and embedded within each therapy rather than as a stand-alone section.

2012

Concepts of wellness and well-being

In fall 2010, staff from NCHS’s cognitive lab performed inductive interviews with 24 volunteers to learn more about individuals who use complementary therapies to promote their wellness, general health and well-being. The interviewer protocol contained questions to elicit reasons or motivations for using complementary therapies without presupposing it was to treat a specific condition. In particular - and in keeping with the published literature [35, 36, 38–40] - respondents were asked about changes noted in “mind or body” when using complementary therapies. Interviewing focused on how respondents who identified “wellness” and “well-being” as their main reason for using complementary therapies understood and conceptualized these terms. Because the intent was to learn more about a general concept or concepts rather than test draft questionnaire items, a more open-ended approach was utilized. When analyzing and interpreting the interviews, researchers aimed for “thick description,” that is, to provide a deeper, more nuanced understanding of the ways that individuals interpret events and phenomena in their social worlds. New participants were recruited and interviewed until no new conceptual insights were identified (saturation). Testing revealed that although respondents reported benefits to their physical health, they primarily associated complementary medicine use with emotional and psychological benefits. The most frequently mentioned benefits were the positive effect on one’s mood, stress level, and overall mental and emotional health. In addition to these emotional and psychological benefits, users also described many different physical benefits, most often mentioned in conjunction with their use of acupuncture, massage, or chiropractic care, and in particular, to help treat and eliminate back pain. The primary components of respondents’ conceptualizations of well-being included being centered, finding balance, and not simply being disease free, but optimally healthy [63].

Based on these qualitative research findings and related literature, a series of items for the 2012 questionnaire was drafted by NCCAM and NCHS staff to capture wellness-related reasons for using complementary therapies. Examples include “because it is natural,” and “because it focuses on the whole person, mind, body, and spirit.” Additionally, some respondents mentioned having benefited from wellness-related outcomes as a result of using complementary medicine. Hence, the questionnaire includes several items asking if using complementary medicine led to outcomes such as “improve your overall health and make you feel better,” “motivate you to eat healthier,” and “help to reduce your stress level or to relax” (Table 8).

Table 8 Examples of wellness-related questions deleted or modified based on cognitive review, focus groups and other input, 2012
Full size table

Draft questionnaire

Twenty-four cognitive interviews were conducted on several different versions of the 2012 draft questionnaire in an iterative process during which revisions were made and retested. Testing focused particularly on two sections: 1) A large set of questions about insurance and payment; and 2) the expanded list of reasons for using the therapy, especially for general wellness and wellbeing. Cognitive testing and focus group feedback (see below) determined that the questions on insurance coverage of, and payment for, complementary therapies were too complex to adequately capture valid information in the limited survey time available. They were replaced with a simple question asking if health insurance covered any of the costs; if yes, did it cover all, some, or none, and how much the respondent paid in total or per visit for the particular therapy [64].

CAMHI conducted focus groups and quantitative analyses

To provide additional input on the 2012 questionnaire, investigators from the Child and Adolescent Health Measurement Initiative (CAMHI) based at the Oregon Health & Sciences University performed quantitative analysis of data from the 2007 child complementary medicine questionnaire including frequencies analysis of the prevalence of each therapy by age group (0–5, 6–11, 12–17), race/ethnicity (Hispanic, white non-Hispanic, black non-Hispanic, other non-Hispanic), disease status, and whether the child used any complementary medicine for reported health conditions [65]. Also explored was how these prevalence rates varied depending on which definition of complementary medicine was used (e.g., whether vitamins and minerals were included in the definition). These analyses led to the identification of sample size deficiencies, appropriateness of given therapies among different age groups, and construct validity of items related to complementary medicine use [65]. Resulting modifications to the 2012 survey included: 1) restricting the child survey to children aged 4 and older; 2) for both the child and adult questionnaires, follow-up questions were not asked for individual traditional healers, movement therapies, and special diets. Rather, individual complementary approaches within each category were grouped to increase cell sizes; and 3) follow-up questions on the reasons children or adults used Ayurveda and chelation therapy were not asked due to small cell sizes on earlier surveys.

In addition, CAHMI organized and conducted key informant interviews and focus group discussions with 21 volunteers from the Family Voices Network [66], as well as from the CAHMI network of expert advisors, pediatric provider partners and family leaders partnering on an NIH/NCCAM grant related to pediatric use of complementary therapies. Informed consent was obtained from all interview and focus group participants. Informant interview and focus group recommendations included: (1) further refinement of the cost and insurance questions, (2) inclusion of child-specific health and well-being reasons for using complementary medicine, (3) the addition of questions about complementary medicine information-seeking, (4) communication with other health care providers about complementary medicine use, and (5) the addition of an item on satisfaction of complementary medicine use for a given reason for use [65]. Although CAMHI-conducted focus groups and quantitative analyses focused on the child component, the provided input contributed to refinement of both the 2012 child and adult questionnaires.

Discussion/conclusions

In this paper, we detail the multifaceted, largely qualitative, iterative process based on literature reviews, expert panel input, and cognitive interviewing methods used to develop and refine the NHIS complementary medicine questionnaires. For the 2012 questionnaire, we also used inductive cognitive review to conceptualize the concept of “well-being” and focus groups with family leaders and subject matter experts to further refine the child questionnaire. By soliciting feedback from subject matter experts, family leaders, and the lay public, we improved the likelihood that the questionnaire items will be understood and interpreted as intended. We describe the systematic evaluation of questionnaire items, the decision-making processes, and rationale for how items were retained, modified, or deleted. By providing this in-depth description of how these influential national surveys were developed, we hope to enable policy makers and researchers to better judge the content validity and utility of the questionnaires and the resultant findings. We also illustrate how the questionnaires have adapted over time to new scientific information and societal shifts so that modifications can be better understood by data users.

There are several limitations to the approach we used to develop the NHIS complementary medicine questionnaires. First, due to time constraints, in many cases, we were not able to continue cognitive testing until we reached saturation and may have missed some problematic issues with the questionnaires that could potentially reduce validity or generalizability. Second, although great care was taken to select individuals for cognitive testing who differed widely in demographic characteristics, the volunteers may not have been representative of the U.S. population in all important respects, which could limit generalizability of the testing results. Third, we did not include lay people on the expert panels convened to determine content areas for the questionnaires and thus could have missed salient topics of particular interest to users of complementary medicine. However, lay public were included in the CAHMI-organized focus groups that identified additional survey topics not prioritized by the think tank experts. Fourth, there is no universally accepted definition of what constitutes a complementary medicine approach. As a result, there is no definitive list of which therapies should be included in the questionnaire. For instance, whether prayer for health reasons should be considered within the realm of complementary therapies has received considerable discussion [67, 68]. For these surveys, we have been as inclusive as possible given time constraints, thus allowing greatest flexibility to data users. Fifth, the NHIS has limitations common to all cross-sectional studies collecting participant-reported data: it is not possible to make inferences about cause and effect, and data are not independently verified. This latter limitation may be particularly important for the section on out-of-pocket expenditures in which respondents likely provide estimates versus exact amounts. Finally, in the 2007 survey and for some respondents in the 2012 survey, although the cost per visit or purchase was collected as continuous data within the range of $0-$500 per expenditure, the number of visits or purchases per person was collected as categorical data (based on recommendations from cognitive testing). This mixing of continuous and categorical data makes the computation of actual expenditures on complementary therapies very complex [69]. For the 2012 survey, we hope to provide better guidance and technical documents explaining how to analyze the expenditure data.

The three NHIS complementary medicine questionnaires were developed to assess the U.S. public’s use of complementary health approaches and factors thought to be associated with this use. It is our expectation that quantitative analyses on the validity and reliability of the questionnaires will supplement the qualitative development approaches describe here, and will inform future survey design and provide guidance to users of existing survey data. Through changes over time, the NHIS complementary medicine surveys have attempted to meet the challenge of providing information to the NCCAM/NIH, national academies, researchers, clinicians, and policy makers. The NHIS complementary medicine datasets remain the primary national source of data on complementary medicine use in the United States.