Background

Information on the use of health care is an essential component of a health information system. Although it is well acknowledged that medical records and administrative data provide the most complete source of information on health care [1], health interview surveys remain an important additional source. First of all, medical records and administrative data are not without problems or inaccuracies [2–4]. Moreover, in many countries administrative and health data are not available in a format that allows producing nationwide information on the use of health care for all population groups. In contrast, survey data provide comprehensive information on a variety of services, and yet, are relatively inexpensive to obtain [5].

The collection of self-reported information on the use of health care is particularly useful for international comparisons. Within the European Union (EU), the need and demand for comparable comprehensive health data and health information has been well recognised [6]. In the past two decades substantial progress has been made towards the development of a permanent health monitoring and reporting system at the EU level [7]. One of the elements in this system is the European Health Interview Survey (EHIS), which includes also a module on use of health care (see Additional file 1). A European Commission regulation [8] guarantees that data obtained through this instrument are presently collected in all EU member states.

The collection of information on the use of health care via health interview surveys has also limitations. Two distinct types of bias may distort the results that are obtained through a population based survey: 1) a non-response or selection bias, if people who participate in the survey have a different consumption pattern than those who do not participate, and 2) a reporting bias, due to memory effects or misclassification by the respondents. Those biases may not only affect the validity of the estimates themselves, but also lead to an incorrect assessment of differences in the use of health care across population groups, e.g. regional differences.

A large body of literature has already investigated the validity of self-reported use of health care. However, many studies focused on specific population groups, were not representative and/or had small sample sizes [3, 9–11]. Some studies were based on national health surveys, but looked at other indicators than the ones used in the EHIS [12, 13]. None of the studies investigated concomitantly the selection and reporting bias or assessed if bias affected the outcome of regional differences in self-reported health care utilization.

In the present study, administrative data from the Belgian compulsory health insurance (BCHI) are used to investigate the validity of self-reported survey information on health care use from the Belgian Health Interview Survey (BHIS) 2008 and to explore to which extent the aforementioned selection bias and reporting bias affect the assessment of regional differences in health care use. Belgium is a country with 3 regions with quite different cultural, socio-economic and morphological characteristics: Flanders, the northern part in which people speak Dutch and which is the wealthiest part of the country, Wallonia, the southern region in which people speak French, and the Brussels Capital Region, which is an exclusively urban region with a large community of migrants and expatriates. The inclusion of EHIS questions in the Belgian national health survey in 2008 enables to explore the validity of regional differences in self-reported use of health care in Belgium for indicators that will be available during the coming years in all EU member states.

Methods

Data

The first dataset included the participants of the BHIS 2008. This survey was conducted between May 2008 and July 2009 among a representative sample of Belgian residents. A detailed description of the study design and sampling methods can be found elsewhere [14]. The response rate of the survey at household level, defined as the number of contacted households that participated in the survey, was 57 %. In total 11,253 people participated in the survey. If the selected person was not able to answer him/herself, a proxy interview was allowed. The study population was restricted to the population aged 15 years and over, resulting in a dataset of 9,651 individuals (BHIS total).

These data were linked by means of a unique identifier (the national number) with data from the BCHI. For the linkage an authorization was obtained from the Belgian Commission for the Protection of Privacy. For 424 individuals (4.4 % of total), the linkage was not possible; those people were consequently excluded from the study. In Belgium the health insurance is compulsory and covers more than 99 % of the population. However, for people working as an independent professional and their dependents (about 10 % of the population), complete coverage was not compulsory before 1 January 2008. For this reason no exhaustive information on reimbursed health care in the year preceding the interview was available for people with an independent profession (and their dependents) who participated in the survey before 1 January 2009 (n = 358). Also these people were removed from the dataset, resulting in a final sample of 8,869 individuals (BHIS linked), consisting of 91.9 % of the initial sample.

The second data set involved a large, unbiased random sample of the BCHI; the sample contains a population which is followed over time. Drop outs due to death and emigration are yearly replaced through a random procedure. A legal framework exists to use these data for policy and research purposes. The database used for this study consisted of the BCHI sample of the people aged 15 years and older for the year 2008 (N = 224,903).

Theoretically it was possible that a person was included in both the BHIS and the BCHI sample. For privacy reasons it was not possible to check this. However, as the probability of such an event was extremely low (roughly 1/40,000), it was assumed that both samples can be considered as mutually independent.

Measures

A first set of measures was survey-based and thus only available in the BHIS. Outcome measures were at least one self-reported contact with a health care service or provider during the past 12 months and, for a contact with a dentist, also the past 6 months. Potential determinants that were explored were gender, age, region of residence, health status, measured through the presence of a chronic disease, illness or handicap, country of birth, education, household type, equivalent household income [15] and information on the person who answered the question (either the selected person him/herself or a proxy respondent).

A second set of measures was register-based and available both in the linked BHIS and the BCHI sample. The way in which the register-based outcome indicators were created, differed slightly by the source. In the linked BHIS the outcome was the prevalence of respondents with at least one registered contact with a health care facility in the 12 months (and for the dentist also in the 6 months) prior to the date of the interview. In the BCHI sample this was the prevalence of people with at least one contact with a health facility during the calendar year 2008. For a contact with the dentist in 6 months, the reference period was a random period of 6 months during the calendar year 2008. Age, gender and region were also available as register-based information. Other register-based measures for which it was estimated that they may have an impact on the validity of self-reported health care were 1) the type of insurance coverage, which essentially indicates whether the person was self-employed (or depended on a self-employed person) or not, 2) whether the person was eligible for preferential reimbursement, which corresponds with a vulnerable socio-economic situation, and 3) the natural logarithm of health care expenses in the year after the survey. The latter is a proxy for the intensity of the use of health care. The health care expenses in the year ‘after’ the survey were used, rather than those in the year ‘before’ the survey, to avoid any interference with the outcome indicators. Natural logarithmic transformation was performed to account for the skewedness of the health cost data. One euro was added to all costs to enable a logarithmic transformation for people who had not incurred any health costs (only 5.7 % of the population).

A final set of outcome measures was based on combined survey-based and register-based information in the linked BHIS. For each type of health care service, variables were constructed indicating whether the BHIS confirmed the information from the BCHI (accurate reporting), generated a false positive result (overreporting), or a false negative result (underreporting).

Analyses

All outcome measures in the study were analysed at the individual level. In a first step, register-based information in the BCHI sample and the linked BHIS was used to assess the selection bias. Next, self-reported and register-based information from the linked BHIS was used to assess the reporting bias. A last set of analyses focused on the exploration of regional differences in the validity of self-reported health care and other determinants.

For each outcome indicator, the prevalence of a contact during the reference period was calculated in several ways: register-based use of health care in the BCHI sample and the linked BHIS, self-reported use of health care in the linked BHIS and the total BHIS. The reporting bias was assessed by calculating in the linked BHIS the proportions of actual agreement between register-based and self-reported use of health care use, overreporting (false positives) and underreporting (false negatives). Concordance was assessed with Cohen’s kappa statistic, which allows the assessment of “agreement beyond chance”. As kappa is affected by the prevalence of the finding under consideration [16, 17] and for rare findings very low values of kappa may not necessarily reflect low rates of overall agreement [18], also the positive and negative predictive values were calculated, as suggested by Cicchetti and Feinstein [19].

Regional variations in the validity of self-reported health care use were first explored via a stratified approach. For each region and each indicator absolute differences in the prevalence of a contact with a health care service/provider and a 95 % confidence interval were calculated according to different scenarios. In a further step, regional differences in the use of health care were investigated with logistic regression models. Age and sex adjusted odds ratios were calculated separately for register-based outcomes in the BCHI sample and the linked BHIS and self-reported outcomes in the total BHIS. By combining the BCHI sample and the total BHIS sample in one dataset, with the source as extra variable and the inclusion of an interaction between source and region in the model, it was possible to test if the observed regional differences varied significantly between self-reported data from the BHIS and register-based data from the BCHI sample.

Regional differences in over- and underreporting were studied by computing relative risk ratios (RRRs) from a multinomial logistic regression model. RRRs refer to the exponentiated coefficients from the model and have to be interpreted as the ratios of two relative risks. E.g., in a multinomial logistic regression model with over-, under- and accurate reporting as dependent variable and region as independent variable, the RRR for overreporting is the ratio between the relative risk of overreporting compared to accurate reporting in region X and the same relative risk in the reference region. In a first model, adjustment was made for age and sex only; in a second model a wide range of other potential confounders were included. Finally, the impact of those other potential confounders on inaccurate reporting, defined as either over- or underreporting, was assessed through a binomial logistic regression.

Analyses were performed with Stata 13.0 [20]. Survey data were analysed taking into account the multistage stratified clustered sampling design of the health survey. In the combined dataset population units from the BCHI sample were given a weight of 1.

Results

Validity of self-reported use of health care

Table 1 presents the distribution by age, gender and region of both the BHIS and the BCHI sample.

Table 1 Basic characteristics of Belgian Health Interview Survey (BHIS) and Belgian Compulsory Health Insurance (BCHI) samples, 2008
Full size table

In Table 2, the comparison of column 1 and 2 assesses the selection bias on the estimates of the self-reported health care use (based on the EHIS questions). For all register-based indicators but one, the linked BHIS yields slightly higher prevalences than the BCHI sample. The relative differences range from 4.2 % to 10.7 %. A lower estimate in the linked BHIS is obtained for an inpatient hospitalisation in the past 12 months, but also here the relative difference is rather small (- 7.6 %).

Table 2 Utilization of various types of health care, by source and assessment method, Belgium, 2008 (population aged 15 years and over)
Full size table

The comparison of self-reported (column 3) and register-based (column 2) outcomes in the linked BHIS allows to assess the reporting bias; it appears that for a contact with a GP and a physiotherapist in the past 12 months, a contact with a dentist in the past 6 months and an inpatient hospitalisation in the past 12 months, the estimates are more or less similar. Relative differences range from -9.1 % to 2.8 %. However, self-reporting strongly overestimates the prevalence of a contact with the dentist in the past 12 months (relative difference of 20.0 %) and strongly underestimates the prevalence of a contact with a specialist (relative difference of -22,3 %) and a day patient hospitalisation (relative difference of -43.9 %) during the same period.

A comparison of column 3 and 4 in Table 2 shows that there is hardly any difference between self-reported estimates in the linked BHIS and the total BHIS.

For a contact with a dentist, a physiotherapist and an inpatient hospitalisation, the overall concordance between reported and register-based use of care is fair to good, with kappas varying from 0.59 to 0.75 (Table 3). Regarding a GP contact, a specialist contact and a day patient hospitalisation, the agreement is moderate (kappa between 0.43 and 0.49). Most striking is the high percentage of false positives (or overreporting) for a contact with a dentist in the past 12 months (14.6 %) and the high percentage of false negatives (or underreporting) for a contact with the specialist in the past 12 months (19.5 %). The latter observation was further explored looking at the type of specialist (data not shown). After adjusting for contacts with other types of specialists, a false negative self-report with a specialist is significantly associated with a contact with a stomatologist (OR 6.15; 95 % CI 2.77–13.63), an ophthalmologist (OR 4.03; 95 % CI 3.00–5.42), a dermatologist (OR 3.30; 95 % CI 2.32–4.68), a psychiatrist (OR 2.12; 95 % CI 1.10–4.08) and an orthopaedic surgeon (OR 2.00; 95 % CI 1.38–2.90) among men, and with a contact with an ophthalmologist (OR 2.06; 95 % CI 1.63–2.61), a pneumologist (OR 1.91; 95 % CI 1.03–3.52), an ENT specialist (OR 1.81; 95 % CI 1.26–2.59) and a gynaecologist (OR 1.41; 95 % CI 1.11–1.78) among women.

Table 3 Concordance between self-reported and registered-based health care utilization (using the latter as reference), Belgian Health Interview Survey 2008 (population aged 15 years and over)
Full size table

The best agreement between reported and register-based use of health care is obtained for an inpatient hospitalisation in the past 12 months: 95.2 % accurate reporting, a kappa of 0.75, a positive predictive value of 75.7 % and a negative predictive value of 97.2 %.

Validity of regional differences in self-reported use of health care

Table 4 shows how age and gender adjusted regional differences in the prevalence of a contact with a health care provider vary in function of the sample (BCHI sample versus BHIS) and the assessment method (register-based versus self-reported). The estimation that can be used as gold standard is the registered information on the use of health care in the BCHI. There is few variation in regional differences in register-based use of health care (column 1 and 2) between the two samples. On the other hand, the assessment of regional differences shows different results for self-reported health care in the BHIS than for register-based use of health care in the BCHI sample. For instance, self-reported survey data appear to underestimate the lower prevalence of a contact with a GP and overestimate the higher prevalence of a contact with the specialist in the Brussels’ Region.

Table 4 Odds ratios for use of health care in Brussels and Wallonia compared to Flanders, after adjustment for differences in age and gender, by source and assessment method, BHIS 2008
Full size table

This is confirmed in Fig. 1, which shows the difference between self-reported use of health care in the BHIS and register-based use of health care in the BCHI sample in the three Belgian regions, disentangling the selection and reporting bias. Figure 1c provides the combined effect of both biases. Regional differences in the validity of the estimate are most pronounced for a contact with a GP and a contact with a specialist. The difference in the prevalence of the population with a contact with the GP in the past 12 months between the self-reported estimate in the BHIS and the register-based estimate in the BCHI sample is smaller in Flanders (-1.8 %; 95 % CI–3.6 %;0.1 %) and Wallonia (-0.9 %;95 % CI–3.0 %;1.1 %) than in the Brussels’ Region (5.0 %; 95 % CI 2.7 %;7.2 %). For a contact with a specialist, the difference is bigger in Flanders (-12.6 %; 95 % CI–14.8 %;–10.3 %) and Wallonia (-11.5 %; 95 % CI -13.7 %;–9.4 %) than in Brussels (-4.5 %; 95 % CI–6.8 %;–2.2 %).

Fig. 1
figure 1

(a) Selection bias (b) Reporting bias (c) Selection and reporting bias, 1Total Belgian Health Interview Sample 2Sample Belgian compulsory health insurance 3Belgian Health Interview Survey Sample for which linkage was possible

Full size image

Table 5 provides information on regional differences of over- and underreporting a contact with a health service. In some cases, over- and underreporting level out, as is for instance the case for differences between Flanders and Wallonia in the prevalence of a contact with the dentist in the past 12 months. In Wallonia significantly more respondents inaccurately report a contact with the dentist in the past 12 months than in Flanders (OR 1.42; 95 % CI 1.16–1.76). At the same time significantly more respondents inaccurately report not to have had a contact with the dentist in Wallonia, compared to Flanders (OR 1.79; 95 % CI 1.31–2.45). As a result the odds ratio of a contact with a dentist in the past 12 months in Wallonia compared to Flanders is quite similar for self-reported information (OR 0.74; 95 % CI 0.64–0.86) as for register-based information (OR 0.70; 95 % CI 0.61–0.81) (Table 4).

Table 5 Regional differences in overreporting a and underreporting b. Results of multinomial logistic regression, Belgian Health Interview Survey 2008 (population aged 15 years and over)
Full size table

Only if regional differences in overreporting are substantially different from regional differences in underreporting, as is for instance the case for a contact with a GP and a specialist in Brussels compared to Flanders, the association between use of health care and region yields different results for self-reported and register-based outcome indicators.

The logistic regression analyses indicate that, also after adjustment for a wide range of potential determinants, regional differences in over and/or underreporting in the use of health care are observed for self-reports of a contact with a GP, a contact with a dentist and a day patient hospitalisation.

Determinants of inaccurate reporting

Table 6 provides information on the factors associated with inaccurate reporting. The association with region confirms the results in Table 5. Women tend to be less inaccurate in reporting a contact with the GP than men, but they are more inaccurate in reporting a contact with the specialist and the dentist. Higher age is associated with less inaccurate reporting of a contact with the GP and the dentist. Age differences in the accuracy to report a hospital admission do no show a consistent pattern. There appears to be no significant association between inaccurate reporting and socio-economic variables, such as education, income and being eligible for preferential reimbursement. The inaccuracy of reporting a contact is also related with chronic disease status and the volume of health care expenses, but the results differ by type of health care provider or service. A proxy interview yields significantly more inaccurate reporting for a contact with the GP and the dentist, but not for a contact with the other health services that are investigated.

Table 6 Determinants of inaccurate reporting a of at least one contact with a health care provider or hospital admission during a reference period. Results of binomial logistic regression (Belgian Health Interview Sample for which linkage was possible - population aged 15 years and over)
Full size table

Discussion

The present study explored the validity of self-reported use of health care in a national health survey, focusing especially on regional differences. The results indicate that compared to administrative data, self-reports in a health survey yield good estimates for the prevalence of a contact with a GP and a physiotherapist and an inpatient hospitalisation; on the other hand, they tend to underestimate the prevalence of a contact with a specialist or a day patient hospitalisation. Self-reporting underestimates the lower prevalence of a contact with a GP and overestimates the higher prevalence of a contact with a specialist in Brussels compared to Flanders.

Although the validity of self-reported health information in a health survey depends both on the selection and the reporting bias, most studies focus on the latter. Generally, validity studies comparing self-reported service use against administrative records show inconsistent findings. Some show a favourable level of congruency between data from the two sources, but others do not [5]. Factors that affect accuracy include age, health status and number of chronic health problems, cognitive abilities, recall time frame, type of utilization, utilization frequency, questionnaire design, mode of data collection and memory aids and probes [1, 2, 21].

An important strength of our study is that it allows assessing concomitantly the selection and reporting bias. The selection bias is rather limited and goes in the same direction for all type of health services, resulting in a slightly higher prevalence of a contact with a health service among survey participants than in the total population. This is in line with a study performed in the Netherlands which concluded that after correcting for differences in demographic variables, respondents and non-respondents differ in the utilization of several types of care, resulting in a small overestimation of utilization [22]. In another study it is reported that the link between health services use and survey non-response may go in different directions [23].

The reporting bias strongly depends on the type of health service that is investigated. It hardly affects the estimation of the year prevalence of a contact with a GP, a physiotherapist, and an inpatient hospitalisation. However, it results into a substantial underestimation of the year prevalence of a consultation with a specialist and day patient hospitalisation, and a serious overestimation of the year prevalence of a consultation with a dentist. Underreporting of a contact with the specialist occurs more among people who had a contact with specific types of specialists, such as a dermatologist, ophthalmologist or gynaecologist. Those specialists are in Belgium often consulted without a referral by a GP and outside a hospital setting. Perhaps this is the reason why there is a recall bias when a contact with a specialist needs to be reported.

Underreporting of a day patient hospitalisation may be due to underreporting of admissions for chemotherapy or kidney dialysis, which are common indications for a day patient hospitalisation, but because of their repetitive character, patients may not conceive this as a hospital admission.

The estimate of a self-reported contact with a dentist appears to be more correct for a reference period of 6 months than for 12 months. This could be due to memory effects. It is also related to the fact that when the reference period of 6 months is used, under- and overreporting level out, whereas for a reference period of 12 months, there is much more overreporting than underreporting. So even if the agreement is not very good, self-reported and register based estimates may be similar if overreporting and underreporting occur to the same extent. Overreporting a contact with the dentist may be related to social desirability, as people may not like to report that they have not consulted a dentist during the past 12 months.

Despite the fact that the study involved multiple testing, we did not apply a Bonferroni correction, as this is a very conservative strategy. Instead, we checked if statistically significant differences were associated with plausible patterns and tendencies.

The concomitant assessment of the selection and reporting bias allows identifying how both biases have an impact on the results. For most types of health services the direction of the biases are opposite to each other, but the reporting bias, predominates. Only for a contact with a dentist in the past 12 months both biases reinforce each towards an important overestimation.

In the present study we investigated the validity of the probability of a contact with a health care service. Many studies focus also on the validity of the quantity of self-reported contacts with a health service [4, 5, 9, 10, 12, 13, 21, 24–29]. This may yield different results. A study in Belgium [12], based on a linkage of data from the health interview survey 1997 with health insurance data, using the number of GP and specialist visits as outcome indicator, found no significant difference between mean self-reported and registered specialist utilization, which is in contradiction with our finding that the prevalence of a contact with a specialist is much lower if it is based on self-reported than on register-based information. This difference could be due the type of indicator (quantity of use versus probability of use), but also to the reference period which was not the same (2 months versus 12 months).

An important focus of this paper is to assess the validity of regional differences in use of health care based on self-reports. One of the core findings in the field of clinical practice variation is that geographical differences in health care utilization and spending are systematic (not just random noise), substantial, pervasive and persistent over time [30]. At the population level, geography has been identified as an important determinant of health care use and health expenditure [31, 32]. Therefore international comparisons on health care use are high on the agenda of international agencies such as the OECD, which produces on a regular basis reports and working papers comparing aspects of the use of health care in its member states [33, 34]. Differences between countries in the organisation of the health system make international comparisons based on administrative and health data difficult or not possible. Therefore geographical differences are often assessed with self-reported data [35, 36].

Previous research concluded that self-reporting offers a reasonably valid estimate of differences in utilization of health care between socioeconomic groups in the general population [37] and has no systematic impact on estimates of ethnic differences in health care utilization [38]. From the present study it appears that, even though the magnitude of the association is not always correctly assessed, self-reported data provide acceptable estimates on regional differences in the use of health care, either because there are no regional differences in over or underreporting, or because regional differences in underreporting and overreporting are of the same size and level out (as it is for instance the case for a contact with a dentist in the past 12 months). However, in that case the study of the determinants of the use of health care, based on self-reported use, will lack validity.

For five outcome indicators we observed a measurement error in the assessment of regional differences in the use of health care due to a reporting bias. Although this does not affect the direction of the associations, it has an impact on the magnitude of the associations, resulting in an over or underestimation of some regional differences. Selection bias only plays a minor role, albeit that the relative overestimation of a contact with a specialist in the Brussels’ Region is the result of both a selection bias and a reporting bias. Although inaccurate reporting is associated with a higher age, chronic disease, a proxy interview and the intensity of health services use, these characteristics do not explain why there are regional differences in over or underreporting. Gender, socio-economic status and country of birth have in our study a limited impact on the validity of self-reported health care use.

The present results are useful for the interpretation of geographical differences in the use of health care based on EHIS data obtained through the same survey instrument. The extrapolation of methodological conclusions from our research to cross-country comparisons in EU member states has of course limitations. In the Belgian context, the organisation of the health care, which has definitely an impact on the health care use, does not vary dramatically across regions, but health systems vary widely between European countries. Moreover, also other methodological aspects such as the place of a question in the questionnaire, mode of data collection, sampling method and recruitment, must be addressed to ensure harmonization in cross country comparisons [39, 40]. Obviously these aspects did not vary by region in the Belgian health survey. Therefore it is quite plausible that European cross-country comparisons in the use of health care will be more affected by validity problems than this is the case for regional differences in Belgium.

Conclusions

The validity of self-reported use of health care, based on EHIS questions, varies by type of health service. Regional differences in the use of self-reported health care may be influenced by regional differences in the validity of the self-reported information.

This finding is important for cross-country comparisons between EU member states, based on the same instrument, especially as cross-country comparisons are more challenging than regional comparisons within one country.

Apart from EHIS, other large scale European surveys, like the European Union Statistics on Income and Living Conditions (EU-SILC) [41] and the Survey of Health, Ageing and Retirement in Europe (SHARE) [42] seek to obtain comparable health data across Europe. A critical reflection on the impact of both the selection and reporting bias on the validity of international comparisons based on survey data, remains important and should be included in the future research agenda.

Abbreviations

EHIS, European health interview survey; BHIS, Belgian health interview survey; BCHI, Belgian compulsory health insurance; EU, European Union; GP, general practitioner; OECD, organisation for economic co-operation and development; EU-SILC, European Union statistics on income and living conditions; SHARE, survey of health, ageing and retirement in Europe; RRR, relative risk ratio