Background

Public health studies need reliable and valid measures of daily food and nutrients intake in adolescents. Among the several methods to assess dietary intake, food frequency questionnaires (FFQs) are commonly used because of their low cost and ease of use [1, 2]. However, the FFQ’s reproducibility is of prime concern [3]. The concept of reproducibility refers to the consistency of data obtained in more than one administration of the same instrument to the same subject at different times [4]. Two types of reliability have been identified, i.e. the relative reliability and the absolute one [5]. Relative reliability is about the consistency of the individual’s position within a group with regards to the others [6]. Basically, food, energy and nutrients intake can vary widely with time, so precision at individual level could be poor even if there is a good agreement of the mean intakes. Therefore, also the absolute reliability, i.e. the degree to which repeated measurements vary for individuals [7], should be taken into account.

Different FFQs have been validated and have been shown to be reliable [4, 8–12], but the need of a web-based, more user-friendly, fast and cost-effective tool has been recently highlighted [13]. To this purpose, the ASSO-FFQ has been developed within the Adolescents and Surveillance System for the Obesity prevention (ASSO) Project, financially supported by the Italian Ministry of Health. It is a web-based questionnaire included in the ASSO-NutFit (Nutrition & Fitness) software that allows obtaining a database on food groups, energy and nutrients intake in adolescents. It has been previously validated against a 7-day weighted food record (WFR) (2014, unpublished observations). The validation study revealed that, even though the ASSO-FFQ was not suitable for measuring the absolute intakes of all food groups and nutrients, it was appropriate for ranking adolescents in food and nutrient levels; moreover, type of school, gender, alcohol consumption and between meals were significant explanatory variables of the intake differences between FFQ and WFR, thus influencing the questionnaire validity.

The aim of this study was to assess the relative and absolute reproducibility of the ASSO-FFQ for 24 food groups, energy and 52 nutrients.

Methods

Study design and participants

This reproducibility study applied a test-retest design. It was approved by the ethical committee of the Azienda Ospedaliera Universitaria Policlinico "Paolo Giaccone" in Palermo (approval code n.9/2011). All participants were provided with information sheets and had to supply the informed consent signed by their parents before the beginning of the study.

A multistage sampling was used for the selection of subjects participating in the Project: at the first stage, a systematic sampling of 7 out of the 55 public and private high schools of Palermo, stratified per type of school (lyceum, technical and professional institute), was performed; in the second stage, a cluster sampling of classes for each selected school allowed obtaining the sample of students. A subgroup of students was selected for the reproducibility study, on the basis of the type of school and age.

ASSO-FFQ’s administration

Participants were asked to web-compile two ASSO-FFQs at one month apart of each other, during classroom time and under the supervision of trained teachers, in March and April 2013. The ASSO-FFQ is a self-administered and semi-quantitative questionnaire, asking the portion size and the frequency of consumption over the previous six months. Portion size is assessed through the use of three pictures showing three sizes of the food/beverage (small, medium, large) and of household units; the following frequencies were used to assess the frequency of consumption: never, 1–2 times per month, once per week, 2–4 times per week, 5–6 times per week, once per day, twice per day, 3–5 times per day.

The ASSO-FFQ comprises a total of 106 food items, and requires on average 20 min to be compiled.

Data collected from both FFQs were processed within the ASSO-NutFit software and were transformed into daily energy and nutrients intake by means of the Italian tables of nutrient composition (http://sito.entecra.it/portale/cra_dati_istituto.php?id=1004&) of the Istituto Nazionale di Ricerca per gli Alimenti e la Nutrizione (INRAN) and of the food composition databases (http://fnic.nal.usda.gov/food-composition) of the United States Department of Agriculture (USDA), that were included into the software.

In order to facilitate the conversion into nutrients, the 106 food items were combined according to their nutrient composition (see Additional file 1) into 24 food/beverage items that were finally investigated: vegetables, fresh fruit, dried fruit, nuts, legumes, breakfast cereals, white bread, bread substitutes, pasta/rice/couscous, potatoes, sweets, cheeses/yogurt, fishery products, meat, eggs, animal fats, oils, savoury food, water, soft drinks, fruit juice, milk, tea/coffee, alcoholic drinks. Energy and a total of 52 nutrient values were also considered as outcomes: total fat, saturated fatty acids (SFA), myristic acid, palmitic acid, stearic acid, monounsaturated fatty acids (MUFA), oleic acid, polyunsaturated fatty acids (PUFA), linoleic acid, linolenic acid, arachidonic acid, eicosapentaenoic acid (EPA), docohexaenoic acid (DHA), trans fatty acids (TFA), cholesterol, proteins, arginine, cystine, phenylalanine, isoleucine, histidine, leucine, lysine, methionine, tyrosine, threonine, tryptophan, valine, carbohydrates, sugar, fructose, lactose, sucrose, starch, fiber, water, calcium, phosphorus, iron, magnesium, vitamin A RAE (Retinol Activity Equivalents), thiamin, riboflavin, niacin, vitamin B6, folate, vitamin B12, vitamin C, vitamin D, vitamin E, ethanol, caffeine.

Further indications on the development, data treatment and validation of the ASSO-FFQ are showed in the validation study (2014, unpublished observations).

Web-based data obtained through the ASSO-FFQ’s compilation were automatically included into a database by the ASSO-NutFit software, after performing an automatic checking of data entry.

Statistical analysis

The obtained database was entered the software STATA/MP 12.1 (StataCorpLP, college Station, TX, USA) and statistical analyses were then performed.

Since the data were not normally distributed, as assessed through the Shapiro-Wilk test, medians and interquartile ranges of food groups, energy and nutrient intakes were carried out on data from the two compiled FFQs. Using the Wilcoxon signed rank test, intake estimates of food groups, energy and nutrients obtained from the FFQs were compared. The proportion of subjects categorized in the same quintile by both the FFQs, in the same or adjacent quintile and in all other quintiles was determined. Weighted kappa was used to express agreement in the classification of individuals and was weighted to take into account the degree of disagreement between the two FFQs. They were compared with the following thresholds [14]: ≤0 = less than chance agreement; 0.01–0.20 = slight agreement; 0.21–0.40 = fair agreement; 0.41–0.60 = moderate agreement; 0.61–0.80 = substantial agreement; 0.81–0.99 = almost perfect agreement.

Intraclass Correlation Coefficient (ICC), one of the most commonly used relative reliability index, was estimated. ICC values were interpreted as follows: ≤ 0.40 = poor reliability; 0.41–0.75 = fair to good reliability; >0.75 = excellent reliability [15].

To describe absolute reliability, Bland-Altman levels of agreement (LOA) were performed according to the following formulation:

d ¯ ± t n - 1 , 0.05 sd 1 + 1 / n

where d ¯ is the mean difference between the FFQs, sd is the standard deviation of the difference between them, t n - 1,0.05 is the value of t corresponding to two-sided p-value = 0.05 for n – 1 degrees of freedom and 1 + 1 / n is an adjustment for small sample size.

The 95% LOA proposed by Bland and Altman were showed to check whether the variability and the precision of the ASSO-FFQ’s measurements were related to the size of the intake estimates [16]. LOA by food groups were obtained overlaying the plot of difference versus mean between the two FFQs. The exponentiated mean difference and LOA provided the ratio of intake estimated by the two FFQs: LOA ranging between 50 and 200% indicated an acceptable agreement [17]. ICC and Bland & Altman analyses were performed on log-transformed, energy-adjusted data to achieve normality, taking into account the confounding effect related to the total consumption of energy. Student t test was used to assess mean differences; significant dependence of the difference in intake estimates from the average level of intake was assessed through linear regression.

Results

Food groups and nutrients intake of 185 male and female adolescents (75% M, 25% F), aged 14–17 (mean 15.9, SD 1.01), was investigated (as shown in Table 1).

Table 1 Sample composition per age and sex
Full size table

Food groups

Food groups’ median intakes, estimated by both the FFQs, are shown in Table 2. Differences between medians were significant (positive) only for bread substitutes, savoury food, water and soft drinks.

Table 2 Median, interquartile range, Wilcoxon test, quintiles comparison, weighted kappa of 24 food groups daily intakes
Full size table

The percentage of adolescents classified into the same quintiles was 40% on average, ranging from 29% (bread substitutes) to 58% (dried fruit), while the percentage of correctly or adjacent classified ranged from 62% (white bread) to 91% (soft drinks), with a mean value of 75%. The weighted kappa values showed substantial agreement (0.61-0.80) for alcoholic drinks, and moderate agreement between 0.41-0.60 for vegetables, fresh fruit, breakfast cereals, bread substitutes, pasta/rice/couscous, potatoes, sweets, cheeses/yogurt, fishery products, meat, eggs, animal fats, oils, savoury food, soft drinks, fruit juice, milk, tea/coffee. Dried fruit, nuts, legumes, white bread and water showed fair values of kappa (between 0.21 and 0.40); no food groups showed low agreement. The mean kappa value was 0.47.

Poor relative reliability was assessed for dried fruit, nuts, legumes, breakfast cereals, white bread, bread substitutes, pasta/rice/couscous, potatoes, fishery products, eggs, oils, with ICC ≤ 0.40, while for all the other food groups, namely vegetables, fresh fruit, sweets, cheeses/yogurt, meat, animal fats, savoury food, water, soft drinks, fruit juice, milk, tea/coffee, alcoholic drinks, fair to good reliability was observed (ICC > 0.40) (Table 3).

Table 3 Intraclass correlation coefficients, exponentiated mean difference and 95% LOA of food groups daily intake, performed on transformed, energy-adjusted data
Full size table

Thirteen out of the 24 food groups showed intake estimates from FFQ2 generally lower than those ones from the first administration; however, these differences were significant (p-value < 0.05) only for meat (Table 3).

The exponentiated value of mean differences (mean ratio) was 100.15% on average. LOA were within 50% and 200% for food groups, except for water, whose lower and upper limits were 39.63% and 262.83% respectively (Table 3).Only eight out of the 24 food groups showed significant dependence (p-value < 0.05) of the difference in intake estimates from the average level of intake: dried fruit, nuts, legumes, bread substitutes, potatoes, meat, savoury food and fruit juice. As an example, scatter plots with LOA of legumes, oils, meat and savoury food are shown (Figure 1).

Figure 1
figure 1

Bland Altman plots for the reproducibility analysis of legumes, oils, meat and savoury food. The solid horizontal lines indicate the mean difference (percentage) between the two measures and the broken horizontal lines indicate the lower and upper Limits of Agreement (±t91;0.025SDs).

Full size image

Energy and nutrients

Table 4 shows median intakes of energy and nutrients, estimated by both the FFQs, and the difference between medians. Wilcoxon signed rank test assessed significant differences of median intakes only for carbohydrates and sugar.

Table 4 Median, interquartile range, Wilcoxon test, quintiles comparison and weighted kappa of energy and nutrients daily intakes
Full size table

The percentage of adolescents classified into the same quintiles was on average 35%, ranging from 28% (sugar) to 56% (ethanol), while the percentage of correctly or adjacent classified ranged from 64% (PUFA) to 90% (ethanol), with a mean value of 72%.

The weighted kappa values showed substantial agreement (0.61-0.80) for DHA, ethanol and caffeine, while ranged between 0.21-0.40 (fair agreement) for energy, total fat, myristic acid, oleic acid, PUFA, linoleic acid, TFA, lysine, carbohydrates, starch, fiber and iron. All the other nutrients showed moderate agreement (between 0.41 and 0.60).

ICC values ranged between 0.21 and 0.40 only for 9 nutrients (total fat, myristic acid, MUFA, oleic acid, PUFA, linoleic acid, cholesterol, starch and iron), while all the other nutrients showed fair to good reliability (ICC > 0.40) (Table 5).

Table 5 Intraclass correlation coefficients, exponentiated mean difference and 95% LOA of nutrients daily intake, performed on transformed, energy-adjusted data
Full size table

For almost all nutrients (48 out of 52) mean differences of intake estimates (FFQ1-FFQ2) were slightly positive, with an average mean ratio of 100.40%; the difference was significant only for carbohydrates and sugar (p < 0.05) (Table 5). LOA were narrow for all nutrients (Table 5), which showed good distribution of the differences in intake estimate around the mean intake.

Arachidonic acid, cholesterol, cystine, carbohydrates, sucrose, starch, fiber, iron, magnesium, thiamine, riboflavin, vitamin B6 and vitamin B12 showed significantly higher differences at lower levels of average intake (p-value < 0.05); on the contrary, difference in DHA intake estimates was lower at lower levels of intake. Figure 2 shows scatter plots with LOA for proteins, total fat, calcium and vitamin E.

Figure 2
figure 2

Bland Altman plots for the reproducibility analysis of proteins, total fat, calcium and vitamin E. The solid horizontal lines indicate the mean difference (percentage) between the two measures and the broken horizontal lines indicate the lower and upper Limits of Agreement (±t91;0.025SDs).

Full size image

Discussion

This study shows the relative and absolute reproducibility of the ASSO-FFQ in estimating 24 food groups, energy and 52 nutrients intake. Results from the analysis indicate that it is a reliable instrument for ranking individuals according to the level of intake.

The reproducibility was estimated by means of different tools. Based on the medians comparison, the intake estimates of all food groups, except for water, soft drinks, bread substitutes and savoury food, were not significantly different between the two FFQs, indicating high reliability of the estimation by the ASSO-FFQ. The results for foods such as milk and cheese, fruit, breakfast cereals, bread, fat spreads, fish/eggs/meat, pasta/rice, potatoes and vegetables are in line with the study from Matthys et al. [18]. The result for water is consistent with a previous study [18], which reported significant difference in the medians of water intake. The low reproducibility found for water in the present study is confirmed also by the kappa value and the percentage of subjects classified in the correct or adjacent quintile, which were among the lowest values obtained (0.32 and 63% respectively); moreover, the analysis on transformed data showed wide LOA, indicating low absolute reproducibility for water. This could be due to difficulties of adolescents in reporting water intake, since it is consumed many times a day and it is difficult to keep count of the right amount consumed. In American adolescents water intake is positively associated with age, and is inversely associated with the intake of beverage moisture and the energy density of foods [19]; maybe a further analysis of the association of water intake with different determinants in our sample could help better understanding water intake.

A similar argumentation could be done for soft drinks, whose consumption has significantly increased in the new generations, often leading them to substitute water intake.

The low reliability of bread substitutes and savoury food could be related to the specific inability of adolescents to count the daily intake of these food groups.The comparison of the mean differences of food intakes indicated also that estimates from the first administration were on average 0.15% higher than those from the second one, thus showing high agreement between the two FFQs. Moreover, all foods showed narrow LOA, indicating a good level of agreement between the two estimates, except for water, as mentioned above. Foods showed in Figure 1 were taken as an example, since results were similar for all foods; a comparison with other studies is not possible since the Bland Altman method has been rarely used in previous reproducibility studies for food intakes.

Also energy and all nutrients did not show significantly different medians, except for carbohydrates and sugar. The result for carbohydrates has been reported previously [20]. However, the relative reproducibility for carbohydrates and sugar was acceptable, as the classification in the same quintile was fair (respectively 30% and 28%), as well as the agreement estimated by the weighted kappa; moreover, ICC values showed fair to good reliability, and the LOA were narrow similarly to those of most nutrients. These results are comparable to those from a previous study [21], which found 36% (for carbohydrates) and 31% (for sugar) of subjects classified in the same quintile and reported high ICC for both nutrients (0.70 and 0.77 respectively). Another similar finding [10] showed 79% of subjects classified in one quintile and a substantial agreement (weighted kappa 0.64) for carbohydrates.

On average, subjects were quite well ranked according to the level of food intake (mean of correctly or adjacently classified 75%, and of correctly classified 40%).

Energy and nutrients showed also good relative reliability, with a percentage of correctly classified subjects ranging between 28% and 56% (mean 35%). These results are slightly higher than those reported previously by Dechamps et al. [21], ranging between 18% and 46% (mean 30%), and Watson et al. [22], ranging between 23% and 39% (mean 32%).

The analysis of weighted kappa showed moderate agreement both for food groups and energy/nutrients (mean kappa values were 0.47 and 0.48 respectively), thus indicating that the ASSO-FFQ has an overall acceptable reliability. Similar results were found for energy and nutrients in other studies [20, 21], which observed mean weighted kappa equal to 0.42 and 0.44. Good results were obtained also with the ICC values, which showed fair/good reliability for most food groups and most nutrients. In line with another study [20] total fat and iron were among the nutrients with poor reliability. On the contrary, a previous study [21] found higher reliability for total fat. However, for these two nutrients in the present study the difference of medians was not significant, the relative reproducibility assessed by the quintiles method was acceptable, the agreement was fair and the LOA were narrow (Figure 2 includes LOA for total fats), thus indicating a reasonable reproducibility.

Bland Altman analysis showed very small mean differences and narrow LOA both for food groups and nutrients, indicating an absolute reliability between the two measures. Moreover, a trend towards smaller difference in some food groups and nutrients according to increased intake values was assessed, so that the level of absolute reliability of the ASSO-FFQ was related to the average level of intake estimates.

Common outcome of many studies [20, 23–25] is that the reported intakes are generally higher in the first administration than in the second. The present study showed that intake estimates from the first administration were on average 0.4% higher than those from the second one. These findings are in line with those studies, even though significant differences were found only for few foods and nutrients.

All the obtained results lead to state that the ASSO-FFQ is a reliable tool. Although gathered measures could be biased by the self-reporting method of the ASSO-FFQ, there is evidence that children are more accurate reporters than their parents [26]. Moreover, as suggested previously [27], the second ASSO-FFQ was administered after one month, an interval that was retained reasonable to avoid change in diet due to food seasonality; other studies considered a longer time interval [8, 22].

However, the study suffers from some limitations. Firstly, the sample was composed of a higher number of male adolescents compared to females; this was due to the predominant presence of males in one of the selected schools. Another limitation was the assumption that for test–retest reliability the true intake did not change between administrations [6]. The other assumption was that the time period between administrations was not too long, in order to avoid any changes in diet or recall bias, and not too short, in order to avoid that subjects could reproduce the answers by mean of learning processes [28].

Conclusions

In conclusion, this reproducibility study provides information on the consistency and stability of the answers of a previously validated FFQ. The ASSO-FFQ is a reliable instrument for estimating food groups, energy and nutrients intake in adolescents, and thus can be used in epidemiological studies on large scale to obtain reliable estimations over time.

Authors’ information

João Breda is staff members of the WHO Regional Office for Europe. The author alone is responsible for the views expressed in this publication and they do not necessarily represent the decisions or the stated policy of the World Health Organization.