FormalPara Key Points

A new bespoke questionnaire (All-MedsNZ) comprising comprehensive data collection elements on natural health products’ (NHPs) and conventional medicines’ use was developed and pretested.

Overall, the questionnaire had good acceptability among a sample of NHP users, and the data collected were of adequate quality to undertake preliminary analyses of prevalence and patterns of NHPs’ use.

Following minor revisions to the questionnaire, future studies should pilot the All-MedsNZ to confirm its feasibility and utility.

1 Introduction

Traditional, complementary and alternative medicine (TCAM), including TCAM products [also described as traditional medicines (TMs)/natural health products (NHPs)/complementary medicines (CMs)], and TCAM practices/therapies (e.g. acupuncture, chiropractic or osteopathic manipulation, reiki) are popular healthcare choices among consumers globally [1, 2]. Many Western countries, including the USA [2, 3], United Kingdom [4] and Australia [5], have nationally representative data on the prevalence of use of TCAM, but only limited data are available for New Zealand (NZ). Several NZ health surveys report data on prevalence of use of TCAM; however, these studies explored selected TCAM approaches and were conducted over a decade ago [6, 7]. These limited data indicate that the use of TCAM in NZ is likely to be substantial, but robust, recent data are not yet available.

At present, TMs/NHPs/CMs [collectively ‘natural health products’ (NHPs)] remain without specific regulations in NZ. Consumers can access NHPs through numerous routes (e.g. pharmacies, grocery/health-food and online stores, natural-health/traditional-medicine practitioners, and others) [8]. Given the easy access to NHPs and their likely substantial (yet largely unrecorded) use, the lack of current information on the prevalence, trends and patterns of use of NHPs, and users’ access to and expenditure on these products, is of concern. Further, if comprehensive data on NHPs’ use can be collected, coded and linked with existing national health datasets, such as medicines dispensing and hospital admissions data, this could (ultimately) provide opportunities to explore associations between NHPs’ use (exposure) and health outcomes. Evidence from data-linked population-based studies (mainly involving conventional medicines) has been shown to influence health policies and enhance the quality of clinical care and patient outcomes [9]. If robust TCAMs exposure data were available in large electronic health datasets through routine collection, it would be possible to explore whether similar impacts can be achieved with respect to TCAMs exposure and health outcomes.

Existing questionnaires that collect data on the use of (exposure to) NHPs [10, 11] have limitations. For example, these tools typically capture use of NHPs by product categories (e.g. ‘herbal supplements’) rather than by seeking detailed information on individual products/preparations used (e.g. HealthBest Ashwagandha dried root ethanolic extract). Also, standardised measurement of TCAM use and definitions for individual terms (e.g. NHPs, TCAM practitioners and practices/therapies) are challenging to apply across countries/regions and may not be universally relevant [12]. Boundaries between TCAM practices/therapies/products and conventional medicines/practices are not static or defined, and differ across countries/regions. Hence, each country needs a measurement tool tailored to its specific context.

Many countries design and utilise national surveys to collect data on the use of NHPs [12], but these are unlikely to translate well across settings, including in NZ. For example, in the USA, an adult and child complementary medicines’ use questionnaire was developed and field-tested in a National Health Interview Survey (NHIS). Throughout different waves of the NHIS, the questionnaire’s content was modified on the basis of an iterative questionnaire development process that included literature reviews, expert panel feedback and cognitive interviewing [13]. Other countries, such as Switzerland, Korea and Taiwan, have also administered their national surveys to explore NHP usage trends [12]. Questionnaires are typically customised to suit the local context. For example, differences in terminology between countries are considered, such as NHPs being known as 'dietary supplements' in the USA [14] and as 'complementary medicines' in Australia [15]. Moreover, regulatory disparities have an impact on questionnaire design, with product registration numbers being incorporated as questionnaire items where regulations exist for NHPs. NZ has a unique indigenous traditional medicine system—rongoā Māori [16]—that is not explored elsewhere. Also, NZ is a bi-cultural country founded on Te Tiriti o Waitangi (the treaty partnership between Māori and non-Māori), but exists as a multicultural society with special recognition of Māori as indigenous people. As such, people from different ethnic and cultural groups living in NZ may use healthcare approaches from traditional medicine systems different to those used in other countries. For these reasons, a bespoke questionnaire is required to collect detailed information on NHPs’ use designed to suit the NZ context and setting. As a result, the questionnaire encompassed response choices such as ‘rongoā Māori’, ‘traditional Māori healer’ and ‘Pacific traditional healer’, in line with the NZ Health Survey [6]. Furthermore, an ‘other’ option was included to account for TCAM products and practices not explicitly listed in the questionnaire.

An international questionnaire - the I-CAM-Q - was developed and published by researchers in 2009 to standardise measurement of broader TCAM use and to allow comparisons across populations [11]. The tool has been translated and adapted across multiple languages and cultures [17,18,19,20,21] with mixed success. The I-CAM-Q was found to have low face validity and acceptability across five European Union (EU) countries: many terms, including types of practitioners, practices/therapies and product categories, were unknown to, or misunderstood by, participants as definitions were not provided in the tool [18]. Several countries adapted the I-CAM-Q by modifying the list of practitioners, practices/therapies and product categories [19, 22, 23], and question response options (e.g. perceived ‘helpfulness’ was changed to ‘benefit’ [24]) to suit their local context; in some countries, changes to the questionnaire were so substantial [17, 25] that the original intention for a standardised tool may have been compromised. In addition, the I-CAM-Q only allowed participants to list up to three TCAM products in four pre-specified categories (‘herbs/herbal medicine’, ‘vitamins/minerals’, ‘homeopathic remedies’ and ‘other supplements’); this approach may miscategorise and/or underestimate the prevalence of use of NHPs, including types of traditional medicines.

The I-CAM-Q also captures the use of TCAMs in isolation: use of conventional medicines is not collected. To understand patterns of use of TCAM, including whether TCAMs are used in addition to, or instead of, conventional medicines, it is essential that both TCAM and conventional medicine use are recorded collectively. Drug interactions between conventional medicines and TCAMs can occur, and some drug–herb interactions have been associated with adverse reactions [26, 27]. Therefore, to answer questions on the prevalence of the use of NHPs, and to fully understand the context of use of NHPs, it is necessary to conduct a nationally representative study and also to explore respondents’ use of TCAM practices/therapies (e.g. acupuncture, chiropractic, etc.) and conventional medicines.

Pretesting is necessary before fielding a questionnaire to assess a tool’s validity and reliability. Cognitive interviewing is a method used to refine questionnaire items and assess validity by exploring item clarity, relevance and participants’ response processes. Typically, cognitive interviews are conducted face-to-face, but this approach has limitations due to small sample sizes, possibly leading to missing or overestimating questionnaire issues or response patterns [28].

Applying cognitive interviewing techniques using web-based questionnaires enables the recruitment of larger and more diverse samples with comprehensive geographic coverage [28, 29]. However, employing web-based administration may result in lower quality data, especially for complex tasks that require participants to explain their responses [29]. To compensate for this, web-based administration can be used to identify obvious problems with questionnaires and then supplemented with in-depth, exploratory, cognitive interviews to provide insight into the more complex aspects of the question evaluation [29].

Thus, this project aimed to develop and test a bespoke questionnaire (All-MedsNZ) comprising comprehensive data collection elements on NHPs’, TCAM practices/therapies’ and conventional medicines’ use. Specifically, the project sought to:

  1. 1.

    Assess the completeness and quality of data provided by NHP users using the bespoke questionnaire

  2. 2.

    Explore the understanding, views and perceptions of users of NHPs in New Zealand on a bespoke questionnaire designed to collect data on the use of NHPs, TCAM practices/therapies and conventional medicines

  3. 3.

    Explore views of NHP users on the design and methods (e.g. web-based mode) of the bespoke questionnaire.

2 Methods

This study utilised two questionnaire pre-testing methods to establish face and content validity: researcher and expert reviews, and cognitive interviewing [30]. Broadly, this project comprised two stages: stage 1 questionnaire development and stage 2 questionnaire testing (Fig. 1). In stage 2, the phase 1 web-based study aimed to identify any questionnaire-related issues across a range of participant demographics. The phase 2 follow-up online interviews aimed to gain further insights into any issues identified in phase 1.

Fig. 1
figure 1

Questionnaire development and testing process. a The expert panel comprised the following practitioners in TCAM and/or conventional medicine: traditional Māori healer, traditional Chinese medicine practitioner, Pacific medicine researcher, homeopath, naturopath and herbalist, general practitioner with interest and practice in integrative medicine, practising pharmacist, Pacific pharmacist, Māori pharmacist

Full size image

2.1 Stage 1: Questionnaire development

Guided by a systematic review of the literature [12] and research team expertise, E.L. drafted and developed a self-administered web-based questionnaire (version 1). The research team conducted a systematic review [12] that focussed on the methods and tools employed in national studies investigating the prevalence of TCAM use in the general population. Based on the findings from this systematic review and the tools used in the NZ Health Survey [6] and NZ Nutrition Survey [7], the All-MedsNZ questionnaire was developed.

2.1.1 Research team and expert panel reviews

Version 1 of the questionnaire underwent two rounds of reviews. First, the research team members (authors), comprising three academic pharmacists, reviewed the questionnaire to ensure it was designed to collect the information needed to meet the research objectives. In addition, the research team checked the questionnaire design and questions, focussing on the terms and wording, structure, response alternatives, order of questions, navigational rules, layout and on identifying typographical errors. The revised questionnaire (version 2) was then reviewed by an expert panel comprising one of each of the following practitioners in TCAM and/or conventional medicine: traditional Māori healer; traditional Chinese medicine practitioner; Pacific medicine researcher; homeopath; naturopath and herbalist; general practitioner with interest and practice in integrative medicine; community/hospital pharmacist; Pacific pharmacist; Māori pharmacist.

The expert panel focussed on the terms and wording of questions and response categories to ensure clarity and applicability to capture participants’ use of NHPs, TCAM therapies and conventional medicines. Based on the feedback from the expert panel, the questionnaire was revised (version 3) and then tested.

2.1.2 Questionnaire content

The questionnaire (version 3) comprised four sections that collected information about participants’:

  1. 1.

    Use of, access to and expenditure on NHPs (42 questions)

  2. 2.

    Visits to and expenditure on natural-health and traditional-medicine practitioners (11 questions)

  3. 3.

    Use of prescription and non-prescription (over-the-counter) medicines, expenditure on non-prescription medicines (10 questions)

  4. 4.

    Demographics (8 questions).

On the instruction page of the questionnaire, participants were informed of the questionnaire’s structure, and that questions were not mandatory to answer. Also, participants were advised to have on hand all the NHPs and conventional medicines they currently take/use to facilitate capture of information and uploading of photograph(s) of each NHP.

2.1.2.1 Operational definitions

At the beginning of the questionnaire, an operational definition for NHPs, including a list of examples, was displayed to participants (Online Resource 1). The operational definition was developed with reference to definitions/descriptions from the World Health Organization (WHO) [31], Cochrane Collaboration [32] and the US National Center for Complementary and Integrative Health (NCCIH). Throughout the questionnaire, the term ‘current use’ of NHPs/medicines was defined as ‘products/preparations/medicines that you are taking daily, or at regular intervals over time (e.g. you take the product once a week), as well as products that you only take when needed (e.g. products for seasonal allergies)’.

In section 3 of the questionnaire participants were asked about conventional medicines they take/use that are prescribed by a health practitioner; health practitioner was described as ‘an authorised prescriber (e.g. general practitioner/family doctor, specialist medical doctor, or other medical/health professional who is legally able to prescribe medicines)’. In the same section, ‘non-prescription medicines’ were defined as ‘also known as “over-the-counter” (OTC) medicines, are medicines that can be obtained from pharmacies and retail outlets, such as supermarkets, without a prescription’.

2.1.2.2 Section 1: Use of, access to and expenditure on NHPs

Participants were asked if they have ever used any NHPs and, if so, to indicate how many different NHPs they currently take/use and had taken/used in the last 12 months. Then, participants were asked if they are currently taking any NHPs that were ‘obtained from a natural health or traditional medicine practitioner/healer’ and any NHPs ‘not obtained from a natural health or traditional medicine practitioner/healer’ and to list the names of the products/preparations, respectively.

For each product ‘obtained from a natural health or traditional medicine practitioner/healer’, participants were asked to indicate the term(s) they would use to describe the type of product, dose form, route of administration, type of TCAM practitioner who recommended the product, cost of the product (or consultation, if the cost of the product itself is not known) the last time they bought the product/saw the practitioner, and how they paid the cost; participants were asked to upload a photograph of the front label of the product (or contents if there is no label) and enter the product’s barcode number (if available).

For each product that was ‘not obtained from a natural health or traditional medicine practitioner/healer practitioner’, participants were asked to indicate the term(s) they would use to describe the product, state the manufacturer/company name, the main ingredient(s) in the product, dosage form, route of administration, where they obtained the product, type of practitioner (TCAM and/or conventional medicine) who recommended the product, cost of the product the last time it was purchased, and how they paid the cost; participants were asked to provide the product’s barcode number (if available) and upload photographs of the product’s front label and ingredient list.

Then, participants were asked if they are currently taking any ‘other’ products/preparations for their health that they had not listed previously and to list the names of these products/preparations; kale powder, pea protein powder and medicinal cannabis were provided as examples. For each ‘other’ product/preparation listed, participants were asked to answer the same set of questions for products ‘not obtained through a practitioner’. In addition, participants were asked why they did not (at least initially) consider this product/preparation to be an NHP.

2.1.2.3 Section 2: Visits to and expenditure on natural health and traditional medicine practitioners

Participants answered questions on the type of TCAM practitioners they had ever met/consulted/had an appointment with, those they had met/consulted/had an appointment with in the last 12 months, and those they are currently meeting/consulting. For each TCAM practitioner that participants had met/consulted/had an appointment with in the last 12 months, participants were asked to indicate the number of times they met/consulted/had an appointment with the specified practitioner in the previous 12 months, the types of treatments/therapies they received, specify if the practitioner is also a conventional medicine practitioner (and, if so, the type of conventional medicine practitioner), the cost of consultation and treatment at their last visit, and how they paid the cost.

2.1.2.4 Section 3: Use of prescription and non-prescription (over-the-counter) medicines, expenditure on non-prescription medicines

In this section, participants were asked if they are currently using any prescribed or non-prescribed/‘over-the-counter’ conventional medicines and, if so, to list the names of these medicines, respectively. For each prescribed conventional medicine, participants were asked to state the proprietary (‘brand’) and/or manufacturer name of the medicine and the dose form. For non-prescribed conventional medicines, participants were asked to indicate the brand and/or manufacturer name, dose form, cost of the medicine the last time they bought it, and how they paid the cost.

2.1.2.5 Section 4: Demographics

Participants were asked questions on their age, sex, ethnicity, location (region of New Zealand and urban/rural area), birth country, father’s birth country and mother’s birth country.

2.2 Stage 2: Questionnaire testing

Questionnaire testing involved two phases:

  • Phase 1: Self-administered web-based questionnaire

  • Phase 2: Follow-up online/telephone interviews.

Based on cognitive interviewing techniques, probes (follow-up questions) were added to the questionnaire (after each question or set of questions) to understand participants’ comprehension of the question(s), recall process of relevant information to answer a question, interpretation of and response to the questions in the questionnaire (Online Resource 2). Participants were asked to rate the clarity of questions on a three-point scale (clear, a bit unclear, very unclear). For questions considered to be ‘a bit unclear’ or ‘very unclear’, participants were asked to describe the difficulties in responding to the particular question or set of questions. Besides clarity, probing questions on the difficulty in answering questions, remembering the costs of the product(s)/preparation(s)/medicine(s)/consultation(s) and types of TCAM practitioners participants have consulted were included (Online Resource 2). These probing questions were displayed only when relevant; for instance, the probing question on difficulties in answering questions about visits to TCAM practitioners was displayed only when participants indicated that they have met/consulted/had an appointment with such a practitioner.

The main focus of the questionnaire was to collect data related to NHPs; because these products are not authorised/specifically regulated in NZ, and because consumers could be taking products purchased overseas, there are inherent difficulties in establishing precisely which products are being described. For these reasons, participants were asked to capture and upload photographs of the product(s)/preparation(s) that they were currently taking. These photographs were used to check the accuracy of the information (e.g. manufacturer/company name, the main ingredient(s), dose form of products/preparations) participants entered/indicated in the questionnaire.

At the end of the questionnaire, other questions sought participants’ feedback on the survey process. Participants rated the survey length, number of questions, the complexity of questions and relative intrusiveness of questions on a scale (from 1—absolutely not acceptable—to 5—highly acceptable). Participants were also asked if they would take part in this survey again and, if not, why. A few open-ended questions were included for participants to comment on taking part in the survey and their thoughts about the questionnaire. Finally, participants were asked if they would prefer to complete the questionnaire in a different language and method (e.g. paper-based/interview instead of web-based).

2.2.1 Participant recruitment and study procedure

Ultimately, the purpose of the All-MedsNZ is to gather data on the prevalence and patterns of TCAM use, including in the context of conventional medicines’ use, in the general population. However, the primary focus of this study was on testing the questionnaire, particularly the items related to NHPs’ use at the granular (i.e. product) level, as they are novel elements. Consequently, the study specifically recruited NHP users to participate in testing the questionnaire.

Participants were self-reported current users of NHPs, aged 18 years and older, living in New Zealand, able to provide consent and complete a questionnaire in English. In addition, participants who volunteered for the phase 2 study were required to have access to the internet/telephone and to be able to be interviewed in English. Data were collected from 1 April 2021 to 9 June 2021.

2.2.1.1 Phase 1: Web-based questionnaire

The phase 1 study aimed to recruit approximately 100 participants. In particular, we aimed to conduct interviews with approximately 30 participants. Anticipating a 30% agreement to participate in the interview (phase 2), we estimated that around 100 individuals would be needed to complete the web-based survey (phase 1) to have around 30 individuals consent to participate in an interview. The study was advertised on the 1 April 2021 through the University of Auckland’s School of Pharmacy and Faculty of Medical and Health Sciences (FMHS) research study recruitment webpages, the FMHS ‘junk mail’ list, which includes academic and general staff and postgraduate students, the study expert panel network, and approximately 1 month of paid Facebook advertising in May 2021.

Potential participants clicking on the web-based questionnaire link in the email invitations and study advertisement were directed to a webpage with a downloadable Participant Information Sheet, and then asked to complete a consent form. The consent form sought participants’ contact details (name, email address and telephone number) if they also volunteered for the phase 2 interview study. After accepting the terms in the consent form, participants were directed to the questionnaire. For completing the questionnaire, participants were offered a NZ$20 shopping voucher that would be posted to them after the end of the study. To receive this, participants were asked to provide a name and address in a separate voucher survey form.

Participants could request a paper-based copy of the questionnaire from the research team. For participants who requested a paper-based copy, a study pack containing the participant information sheet, consent form, study questionnaire, voucher survey form and three pre-paid addressed envelopes for returning the questionnaire, signed consent form and completed voucher survey form were mailed to these participants.

2.2.1.2 Phase 2: Online interview

A list of participants interested in being interviewed was obtained from responses on the consent forms. The study aimed to contact up to 30 people via telephone or email from this list. During the phase 1 recruitment period, at the end of each week, the demographics of participants who had consented to a phase 2 interview were tracked. Interview participants were selected across several demographic variables: age groups (18–25 years, 26–64 years, 65+ years), sex (male, female), ethnicity (NZ European, Māori, Pacific, Chinese, Indian, Other) and location (urban/rural), and users of various types of NHPs where possible.

E.L. contacted selected participants to arrange a mutually acceptable date and time for an interview. Interviews took place online through Zoom, a video communication platform, and were audio recorded. Before the interview, participants were emailed a copy of their responses to the web-based questionnaire. Those responses, with an interview schedule, were used to guide the interview discussion. Interviewed participants were offered a NZ$50 store voucher.

E.L. transcribed the interviews; J.B. checked 10% of the transcripts for accuracy. On the consent form, participants indicated their choice to review their interview transcripts. Where participants accepted this offer, they were emailed a password-protected copy of their transcript and invited to make/highlight their changes and return the transcript to the study team within 2 weeks of receipt. If transcripts were not returned, it was assumed that the participant did not wish to make changes.

2.3 Data analysis

Data from the web-based questionnaire were exported from Qualtrics to Microsoft Excel in Office 365 v16.75.2 (Seattle, USA) for conducting descriptive analysis. Focussing on the NHPs-related questions, question response rates were reviewed to indicate those that were unanswered by a proportion of participants. Then, the completeness and quality of information provided by participants were assessed by analysing a subset of data (data on NHPs ‘not obtained from a natural health or traditional medicine practitioner/healer practitioner’).

To assess completeness of information provided by participants, the types of responses given (whether they were what the question intended to collect) were assessed. For each NHP recorded in the data subset, participants’ answers to questions on product name, manufacturer/company name, the main ingredient(s), dosage form and barcode number were checked for correctness of the type of information entered.

To assess data quality, the product information entered was checked against the product photographs uploaded by participants. Each product was scored from 0 to 4, where 1 mark was given for each correctly entered product brand/generic name (e.g. multivitamin and mineral boost), manufacturer name (e.g. HealthyWay), the main ingredient(s) (e.g. multivitamin and multimineral), and dose form (e.g. capsules). For the main ingredient(s) criteria, the research team developed and applied the following rules:

  • All main ingredient(s) or categories of ingredient(s) must be stated; for example, ‘probiotics’ (category) and ‘Lactobacillus acidophilus M23’ (individual ingredient) are accepted as correct

  • No marks were given if one or more wrong ingredient(s) was entered

  • For multivitamin products, ‘multivitamin(s)’ was accepted as correct.

Barcodes were searched in two databases [33, 34] to verify if the barcode matched the product name.

Participants answered a question on the ‘terms’ they would use to describe each product (Box 1). For products where only ‘vitamin(s) and/or mineral(s)’ was selected, the ingredients in the products were checked using photographs uploaded by participants to determine whether or not any non-vitamin and/or non-mineral ingredients were listed on the label.

The chi-squared test of independence was used to determine whether significant differences exist between correctly stated main ingredient(s) and product type (single or multi-ingredient). Fisher’s exact test was used to determine if there were statistically significant differences between ‘products correctly described as a vitamin(s) and/or mineral(s) only’ and type of product (single or multi-ingredient). Inferential statistics were conducted using R 4.0.3, and the significance level was set at p < 0.05.

Box 1 Question about terms used to describe types of natural health products/preparations
Full size table

Responses to the probing (follow-up) questions were analysed to identify issues with comprehension, recall and judgement. In addition, participants’ views regarding the study methods (preference for language, questionnaire delivery method) were summarised. Open-ended questions were collated and selected participants’ responses were reported verbatim.

As the intent of the interviews was to elicit information on the relevance and clarity of the questionnaire items (questions), the individual item (as opposed to the individual participant) was regarded as the unit of analysis.

Findings from phases 1 and 2 were collated and used to guide the research team’s decisions about keeping, deleting, or modifying the questions in the questionnaire. The proposed changes to the questionnaire were shared with the expert panel for feedback.

2.4 Ethics approval

The questionnaire testing study was approved by the University of Auckland Human Participants Ethics Committee on 08/03/2021 for three years, with reference number UAHPEC2550.

3 Results

3.1 Expert panel review

Feedback was obtained from all members of the expert panel. Overall, the questionnaire had high content validity and most of the original questions were retained. Some parts of the questionnaire were modified on the basis of suggestions from the expert panel. This included modifying the definition of several terms; for instance, the term ‘traditional Māori medicines’ was revised from ‘native plant-based preparations (rongoā rakau)’ to ‘preparations usually made from certain plants, minerals, animal products and/or other substances’. Under the NHP definition, product categories were added (e.g. specially compounded formulations) and revised (e.g. separating definitions for homeopathic and flower remedies), and examples were added for the different product categories to help improve participants’ understanding of the terminologies used. Where participants were asked to indicate the number of different NHPs they take, photographs were included to help participants determine what is considered to be one product (e.g. a preparation containing a mixture of crude herbs is one product).

Initially, participants were asked to list the names of all products/preparations they currently take as a single question; subsequently, this question was split to ask participants to list the names of products they obtained from natural health and/or traditional medicine practitioners and those obtained elsewhere. This change was suggested as some of the subsequent questions [e.g. manufacturer name and main ingredient(s)] may not be relevant for products/preparations obtained from natural-health and/or traditional medicine practitioners.

For visits to TCAM practitioners, a question on the types of treatment/therapies received by participants was added, because multiple treatments/therapies may be provided by TCAM practitioners (e.g. naturopaths may provide lifestyle/dietary advice and/or prescribe herbal medicines and/or homeopathic remedies).

Other minor changes were incorporated into the questionnaire, such as rewording some questions to improve clarity. For example, ‘Have you ever consulted any of the following practitioners for your own health?’ was revised to ‘Have you ever met/consulted/had an appointment with any of the following practitioners for your own health?’. For several close-ended questions, additional response choices were included; for instance, ‘Work and Income New Zealand (WINZ)’ and ‘koha’ (a New Zealand Māori term for gift, present, offering, donation, contribution—especially one maintaining social relationships—and has connotations of reciprocity) were added as choices for the question ‘How did you pay the cost?’.

Several questions were removed from the questionnaire. For example, questions regarding health reason(s) for taking individual NHPs and conventional medicines and the dose, frequency and duration of use for each NHP and conventional medicine named were removed. These questions are relevant and provide useful information about exposures to NHPs/conventional medicines but increased the length of the questionnaire and respondent burden. The removal of these questions did not affect the primary purpose of the questionnaire, which is to collect comprehensive data on the prevalence of use of TCAM in New Zealand.

Modifications to study methods comprised inclusion of a paper-based option for completing the questionnaire to facilitate response and minimise under coverage, particularly among the older, rural population with limited internet access and digital literacy. Participant remuneration was revised from offering a prize draw to offering a NZ$20 shopping voucher for each participant in the phase 1 study as this was deemed more appropriate considering the length of the questionnaire and to encourage response.

3.2 Questionnaire testing

In total, 95 and 27 participants completed the phase 1 and phase 2 studies, respectively. All participants completed the web-based version of the questionnaire. The median time taken to complete the questionnaire was 25 minutes [interquartile range (IQR) 16–39 min]. Interviews ranged from 20 to 48 min in duration. Participants were mostly 20–39 years old, female, NZ European and living in Auckland (Table 1).

Table 1 Participants’ characteristics for the phase 1 (n = 95) and phase 2 (n = 27) studies
Full size table

Most participants (n = 80) obtained their NHPs through self-selection (i.e. not through a natural health/traditional medicine practitioner). More than half of the participants (n = 53) were currently taking prescribed conventional medicines, and approximately one-third (n = 37) were currently using non-prescription/‘over-the-counter’ conventional medicines. Over two-thirds (n = 72) of NHP users also used conventional medicines. About 40% of the participants met/consulted/had an appointment with one or more natural health and/or traditional medicine practitioners in the previous 12 months (Table 2).

Table 2 Participants’ use of natural health products and conventional medicines, and visits to natural health and/or traditional medicine practitioners
Full size table

Overall, 97 and 265 entries for NHPs that were obtained from a natural health or traditional medicine practitioner/healer or through self-selection were recorded, respectively. Additionally, 40 ‘other’ products/preparations were recorded. In total, 141 entries for prescribed conventional medicines and 64 for non-prescribed medicines were listed by participants. For consultations with TCAM practitioners, 86 entries were recorded.

3.2.1 Phase 1 study: Response rates

Most questions had a response rate ≥ 90%, except for several questions in sections 1 to 3 of the questionnaire (Table 3). Almost all questions with low response rates were open-ended (rather than multiple-choice). Few participants did not provide information about manufacturer name and main ingredient(s) of product(s)/preparation(s)/medicine(s). Question response rates < 62% were mainly related to cost, and providing barcodes and photographs of products/preparations (highlighted rows in Table 3).

Table 3 Questions with a response rate below 90%
Full size table

3.2.2 Phase 1 study: Data completeness

Of the 265 NHPs that were ‘not obtained from a natural health or traditional medicine practitioner/healer’, > 80% had the correct type of product information entered (Table 4). The relatively high percentage (10–15%) of blank responses in the manufacturer/company name and main ingredient(s) questions was because some participants had included this information (60% and 87% of blank responses, respectively) in other columns (e.g. product brand/generic name) instead.

Table 4 Review of the type of product information entered by participants (n = 265)
Full size table

From the total NHPs recorded (n = 265), only 137 (51.7%) had a barcode entered and 113 (82.5%) of these were valid (e.g. correct length).

3.2.3 Phase 1 study: Data quality

Overall, 143 (55.9%) NHP entries had photographs of both the front label and ingredient list uploaded; of these, 58 (40.6%) and 85 (59.4%) were single- and multi-ingredient products, respectively. Since participants answered specific product information (e.g. manufacturer name) in other columns (e.g. product name), data quality was assessed and scored on the basis of the product information entered in totality, regardless of whether the information was recorded in the correct column (Box 2). Overall, most products had a score ≥ 3 (Table 5); of the products with a score of 3, 53 (89.8%) were not given a full score due to the main ingredient(s) information being missing or incorrect. Multi-ingredient products were more likely to have the main ingredient(s) information missing or incorrectly stated (χ2 = 53.4; df = 1, p < 0.01) compared with single-ingredient products.

Table 5 Data quality assessment (n = 143)
Full size table

Approximately half (45.5%) of 33 products were incorrectly described by participants as being ‘vitamin(s) and/or mineral(s)’ only. These products included other ingredients, such as herbs (e.g. echinacea, rose hip, ashwagandha), amino acids, collagen and coenzyme Q10. When compared with single-ingredient products, multi-ingredient products were more likely to be incorrectly described as ‘vitamin(s) and mineral(s) only’ (Fisher’s exact test; p < 0.01).

Of the 113 valid barcodes, 70 (61.9%) matched the product entered; the remaining barcodes did not match the product, or were not found in the barcode database search.

Box 2 Example of scores given on the basis of the product information entered
Full size table

3.2.4 Phase 1 study: Responses to probing questions

Overall, participants had no major problems understanding and answering questions in the questionnaire; some minor issues were identified (highlighted items in Table 6).

Table 6 Participants’ responses to probing questions in the web-based questionnaire
Full size table

Over 85% of participants rated the questions as ‘clear’ and had no problems understanding the questions. Questions that were rated ‘a bit unclear’ and ‘very unclear’ (by ~ 8% of participants) were related to listing the names of NHPs the participants were currently taking (questions 4–7) and answering a set of questions about participants’ visits to TCAM practitioners over the past 12 months (questions 48–52). From open-ended text responses, few participants stated difficulties in differentiating products they obtained and did not obtain from a natural health or traditional medicine practitioner/healer. Some participants were unsure of the definition of the term ‘natural health practitioner’ used in this study.

Generally, most participants did not have problems answering questions in the questionnaire. Questions that were rated problematic to answer (by ~ 20% of participants) were the set on participants’ use of NHPs ‘obtained from a natural health or traditional medicine practitioner/healer’ (questions 8–15), ‘not obtained from a natural-health or traditional-medicine practitioner/healer’ (questions 16–27) and use of non-prescribed/‘over-the-counter’ conventional medicines (questions 60–63). A few participants described their problems with answering these questions; the reasons included difficulties in remembering the cost of products/medicines and not being able to upload photographs as they did not have the products/medicines during completion of the questionnaire.

Approximately half and one-third of the participants had difficulty remembering the costs of their NHPs and non-prescribed/‘over-the-counter’ conventional medicines, respectively. Some participants were unable to remember the cost of each product/medicine because these products/medicines are not kept in their original packaging with the price tag, were purchased some time ago, or paid for together with other products/medicines.

For participants taking ‘other’ products/preparations (besides the NHPs obtained and not obtained from a natural health or traditional medicine practitioner/healer), some explained that they considered the particular product/preparation (e.g. chia seeds, protein powders, collagen powders) a ‘food’ rather than an NHP or were not sure if the product/preparation is an NHP. Many other participants actually considered these ‘other’ product(s)/preparation(s) an NHP, but had forgotten/overlooked the product(s)/preparation(s) when completing the previous questions in the questionnaire.

3.2.5 Phase 1 study: Feedback on the questionnaire

Most participants rated the survey length, number, complexity and relative intrusiveness of questions acceptable (Table 7). Two (2.11%) participants indicated that they would not take part in this survey again; the reason given was ‘survey was too repetitive’ and ‘took too long’. Three participants would prefer to complete the questionnaire in a different language: two stated Chinese/mandarin, one stated Spanish. Four participants would prefer to complete the questionnaire using a different method: two preferred an interview, one preferred using a computer rather than a mobile telephone and one did not state the method desired.

Table 7 Participants’ rating of the web-based questionnaire
Full size table

In open-ended questions, 59 and 25 participants provided their thoughts about the questionnaire and comments about participating in the study, respectively. In general, participants had positive experiences with the questionnaire and study and did not face major issues with completing the questionnaire. A few participants raised some minor problems related to the length of the questionnaire, difficulties with cost-related questions, uploading photographs of NHPs and clarity regarding how some terminologies used in the questionnaire relate to different contexts of access (e.g. are vitamins prescribed by a general practitioner considered natural health products?).

3.2.6 Phase 2: Qualitative interview data

Data from the interviews reiterated the main issues found from the phase 1 study and further explained the problems with the questionnaire. Selected quotes from participants are included in Online Resource 3.

One issue was that participants did not have all the NHPs and conventional medicines on hand during completion of the questionnaire; this partly contributed to the low response rates for some questions (e.g. the main ingredient(s) and uploading photographs). Some participants were unaware that a statement recommending participants to record all their NHPs and conventional medicines they currently take was displayed at the beginning of the questionnaire.

The term ‘natural health practitioner’ was unclear for some participants where they were unable to distinguish if a practitioner (e.g. a nutritionist) is a TCAM or conventional medicine practitioner. Some participants did not remember the type(s) of practitioners they had consulted, hence, they were unsure if they had visited a ‘natural health practitioner’.

Differentiating the NHPs obtained (question 5) and not obtained (question 7) from natural health or traditional medicine practitioners/healers was challenging for some participants. Some NHPs, like vitamins and minerals, may be prescribed by conventional medicine practitioners and participants were unsure if these NHPs should be included in their response to question 7. For some participants who were natural health or traditional medicine practitioners themselves, these questions become confusing when the NHPs they take are self-prescribed.

Interpretation of question 28 (‘Are you currently taking/using any other natural health-type product(s)/preparation(s) for your health that you have not listed previously?’) differed between participants. Some participants understood the question as asking for products/preparations considered ‘food’ rather than NHPs. In contrast, others felt the question was a reminder to include NHPs they may have forgotten to list earlier. For some participants, going through prior questions allowed them to gauge the scope and context of the questionnaire, and the ‘other’ products/preparation question provided them with an avenue to include NHPs they had not mentioned.

Listing the main ingredient(s) of products/preparations was also a challenge for some participants. Some participants were unaware of the ingredient(s) in the products/preparations they took, or the list of ingredients on the product label was too long to include in the questionnaire. For some participants, the product ingredients information entered was incomplete. One participant who entered one ingredient only for a multi-ingredient product explained that the product was used for the purpose of one of its ingredients only (e.g. biotin), although it is a multi-ingredient product containing other ingredients (e.g. marine collagen, horsetail).

There were two main reasons for the low response rates for questions related to manufacturer/proprietary (brand) name and providing barcodes: participants had removed the product from its original packaging, or the information was not available on the product label. Similarly, some participants could not provide photographs of NHPs as the original packaging was no longer available to them. Other participants were unaware of the questionnaire instructions stating that photographs of NHPs would be required; hence, they were unprepared to submit photographs (e.g. did not have products available and/or completed the survey away from their home). A few participants did not provide photographs because of the inconvenience and, for some, technical complexity of doing so.

Interviews with participants revealed that questions on costs (‘About how much was the cost the last time you bought this product/preparation/medicine?’) were particularly difficult to answer for various reasons. For product(s)/preparation(s)/treatment(s) obtained from a practitioner, the cost may not be known to participants, as payments are typically made together with consultation fees. Some participants could not remember the costs of products/medicines because they purchased them some time ago and/or products did not have a price tag. Participants also said that cost would differ based on the product pack size (e.g. 30 versus 60 capsules) they had chosen to purchase (participants were asked to state the cost the last time they bought the product); this has implications for the interpretation of the data collected on costs of NHPs. One way to mitigate this is to ask participants additional questions regarding the cost of a product/preparation/medicine, which was explored during the interviews. Participants were asked if they could estimate how long the product/preparation would last them; most were able to answer without much difficulty (Online Resource 4). It was more difficult for participants to answer this where products are not taken regularly; however, they were still able to provide an estimated duration of use. Alternatively, participants were also asked if they were able to estimate their average total spending per month on NHPs; most participants found this challenging because, usually, not all products they are taking were purchased at the same time.

3.3 Expert panel and research team reviews

A summary of results, including the revisions made to the questionnaire, was sent to the expert panel; no further comments were received. Following input from a Māori research team member, several questions exploring barriers to TCAM access were also included on the basis that the data collected from these questions would provide some understanding of the demand for TCAM and, to some extent, explain (the low) prevalence of use for some categories of TCAM (e.g. types of traditional medicine).

3.4 Summary of changes to the questionnaire

Overall, no major study methods or questionnaire-related issues were identified. Based on the minor issues found in the questionnaire testing study and reviews from the expert panel and research team members, parts of the questionnaire were revised (final version) (Online Resource 5). Changes made to the questionnaire are available in Online Resource 6. The main changes are described below.

Due to the uncertainty about separating and listing the names of NHPs obtained and not obtained from a ‘natural health or traditional medicine practitioner/healer’ [i.e. where to include products (e.g. iron tablets) prescribed by a conventional health practitioner, such as a general practitioner?], the questions were changed to ‘formulated or specially compounded products/preparations’ and ‘manufactured/commercial natural health or traditional medicine products/preparations’. This change would allow participants to list all types of products, including those prescribed by conventional medicine practitioners.

The cost question was revised to asking participants to estimate the cost to the nearest NZ$5, specifying ‘for one unit/bottle/box/packet of the product/medicine’. In addition to asking about the cost of a particular product/preparation/medicine, we added questions on the date of last purchase and the length of time that one unit/bottle/box/packet of purchased product/medicine would last the respondent.

Despite the problems (low response rates, inaccuracies with information entered) with the questions on the barcodes, main ingredient(s) and uploading photographs, these questions were retained in the questionnaire. This is because barcodes and photographs, where provided by participants, are useful to check the accuracy of information [including main ingredient(s)] entered by participants.

Several questions exploring barriers to TCAM access were added; these questions were adapted from the New Zealand Health Survey [35]. Questions asking participants if they had a medical problem or TCAM prescription/recommendation in the last 12 months, but did not visit a TCAM practitioner or purchase recommended TCAM products/preparations, respectively, because of cost or other reasons, were added.

4 Discussion

This project developed and tested a bespoke questionnaire (All-MedsNZ) that included comprehensive data collection elements relating to NHPs’, TCAM practices/therapies’ and conventional medicines’ use. Findings from this project were used to guide refinement of the questionnaire and methods for a future, nationally representative study on NHPs’, TCAM therapies and conventional medicines’ use in NZ. To our knowledge, this is the first study in NZ that explored self-reported TCAM data at a granular (e.g. specific TCAM product) level. We argue that it is essential to obtain comprehensive TCAM use data at this level rather than merely a category level (e.g. ‘herbal medicines’) directly from participants. However, we acknowledge the inherent trade-off between data quality and completeness, as the collection of information at such a detailed level can impose a higher burden on respondents. Nevertheless, the findings from this testing study provide valuable insights into the complexities in collecting TCAM data in NZ and serve as a basis for future research in this field.

The development, testing and refinement of All-MedsNZ were based on systematic literature reviews, expert panel input and cognitive interviewing methods, which are widely-used questionnaire pretesting methods, including by the US NHIS [13]. The contents of this study’s questionnaire were validated by an expert panel encompassing TCAM and conventional medicine health practitioners. During the testing phase, the questionnaire had good acceptability among participants (NHP users). Most participants completed the questionnaire and indicated they had no problems understanding and answering the questions.

Involving consumers/patients in designing a new questionnaire has been shown to improve questionnaire comprehensiveness and relevance while reducing item ambiguities [36, 37]. Some participants found questions related to cost challenging to answer: 8–48% indicated problems remembering the costs of products/preparations/medicines/visits to practitioners. Participants had difficulties answering the original cost question because prices of products/preparations/medicines are dependent on the purchased pack size (e.g. 30 versus 60 capsules/bottle). The two-phase testing adopted in this study permits further examination of the issue during the phase 2 online interviews, elucidating the various reasons why the cost questions were difficult to answer, which is difficult to explore within the questionnaire. In addition, the qualitative nature of the interviews allowed exploration of participants’ thoughts about potential revisions to the cost questions. In previous studies, the cost of TCAM was explored using various methods. The US NHIS asked respondents to provide the cost the last time they (respondent) purchased NHPs (e.g. vitamins and minerals) and how often these NHPs were bought. For TCAM practitioner visits, the ‘average’ amount spent per visit, or total cost over 12 months, was collected [38]. In Australia, respondents were asked to estimate, to the nearest dollar, the monthly cost of NHPs purchased and the approximate total yearly cost of visits to TCAM practitioners [39]. In the phase 2 interviews in the present study, participants were asked about the ease of estimating the cost of each NHP and their average monthly spending on NHPs. Subsequently, the questionnaire was revised, and several other cost-related questions (date of last purchase and the length of time that a purchased product would last the participant) were added to the questionnaire.

Another issue concerned asking participants to list the main ingredient(s) of NHPs, particularly for multi-ingredient products. About one-third of the 143 NHPs for which photographs of the product’s front label and ingredient list were provided had one or more of the main ingredient(s) missing or incorrectly stated in the self-reported data. Some participants were unaware of the ingredient(s); for some NHPs, the ingredients list was too long to list in the questionnaire. Also, some participants listed particular ingredient(s) of interest while disregarding other (potentially active) NHP ingredient(s). Hence, consumers may not reliably self-report the ingredient(s) of the NHPs they take, especially for multi-ingredient products. While this may not affect broad-level analyses (e.g. the prevalence of use of NHPs in a population), there are implications for more defined analyses at the NHP category/product level (e.g. the prevalence of use of herbal medicines or products containing ‘echinacea’). To achieve data accuracy for such defined analyses, product barcodes and photographs uploaded by participants can be used to cross-check lists of ingredient(s) entered by participants. However, this approach is resource intensive for researchers and (ultimately) for curators of real-world health datasets that include data on TCAM exposures. From participants’ perspectives, entering barcode numbers and uploading photographs is burdensome; low response rates for these items were observed in this study (Table 3). In some countries, such as Australia [40] and the UK [41], where (many) TCAMs are regulated, product registration numbers can be collected; this would alleviate the need for participants to transfer information provided on product labels to a questionnaire. Currently, there are no regulations or legislation specific to NHPs in NZ. In July 2023, the proposed Therapeutic Products Bill received Royal Assent, becoming the Therapeutic Products Act (2023) [42]. This Act will include regulations for NHPs. The regulations will aim to ensure acceptable safety and quality of NHPs, although full details of how these will be assured are not yet known [42]. The All-MedsNZ questionnaire may require revision following implementation of regulations proposed by the Bill. It is also important to consider that the proposed bill may not capture all NHPs; for instance, individuals could continue to import products from overseas for personal use.

Approximately half of the 33 NHPs described as ‘vitamin(s) and mineral(s) only’ actually were labelled as containing non-vitamin and/or non-mineral ingredients (e.g. herbs, amino acids) when the individual product’s ingredient list was checked using photographs uploaded by participants. Despite the small sample size, this finding indicates that relying on consumers to self-report the use of NHPs in pre-specified categories is not feasible. This issue also arose in the implementation of the I-CAM-Q, where respondents declared their use of NHPs in four pre-specified categories (‘herbs/herbal medicine’, ‘vitamins/minerals’, ‘homeopathic remedies’ and ‘other supplements’): considerable data cleaning was required in one study to recode NHPs reported by participants to the correct category [43]. Another concern, particularly with multi-ingredient NHPs, is that these products (e.g. vitamin C plus echinacea) may fall into more than one category (e.g. vitamins and herbals). Although using pre-specified categories may cue respondents to remember the products they take, collecting data in such a manner is error-prone and may underestimate the use of NHPs.

This study has several limitations. More broadly, a universal definition for TCAM and what constitutes a TCAM approach does not exist. Hence, as with all TCAM studies, findings depend on respondents’ understanding of terminologies (e.g. ‘NHPs’), which may vary considerably across individuals. To ensure consistent interpretation of terms among participants, comprehensive operational definitions/descriptions of terminologies are needed and were used in this study. Still, participants’ awareness, understanding and use of these definitions in providing their responses is not guaranteed. We acknowledge the potential for subjectivity in the participants’ responses, as well as the influence of recall bias in their reporting. These factors should be taken into consideration when interpreting the findings of this research.

Beyond TCAM, this work also raises methodological issues. Web-based studies are challenging, especially among the older, rural population who have limited access to the internet and low(er) digital literacy. The focus of this study was to test a questionnaire that is self-administered through the web, which is known to be a convenient and cost-efficient delivery method. However, this mode of data collection must be weighed against the risk of population under coverage. Hence, paper-based copies of the questionnaire were also offered to participants in this study. No request for a paper-based questionnaire was received. Therefore, findings from this study are limited to individuals with internet access and who are technology literate.

In this project, the sample of study participants was not intended to represent the general population in NZ, rather to provide depth and breadth in shared perspectives. Although care was taken to select individuals for the online interviews who differed in demographic characteristics, the choice of sample was limited by the demographics of those who had completed the questionnaire and was clearly not nationally representative. Therefore, the findings from this study cannot be generalised. A larger, nationally representative pilot study should be conducted to explore the feasibility and utility of the revised All-MedsNZ (Online Resource 1). Also, it is expected that future quantitative analyses on the validity and reliability of the questionnaire will supplement the work described in this pre-testing study to inform further refinements of the questionnaire.

5 Conclusion

Overall, no major design-, method- or questionnaire-related problems were identified in the testing study of the All-MedsNZ. This bespoke questionnaire collects comprehensive data relating to NHPs’ and conventional medicines’ use. The questionnaire had adequate face and content validity and acceptability among participants, and most of the data collected on NHPs’ use is complete and of sufficient quality for analysis. The questionnaire was revised to enhance the clarity of several questions and to mitigate some of the minor issues with the questions on costs. The final questionnaire should be piloted in a larger, nationally representative study to confirm its feasibility and utility.