Use of online and paper-and-pencil questionnaires to assess the distribution of orthorexia nervosa, muscle dysmorphia and eating disorders among university students: can different approaches lead to different results?



Administration of questionnaires to assess the diffusion of disordered eating behaviours via the web is becoming common today. The aim of this study is to assess whether two different approaches of administering a test to assess traits of eating disorders (EDs), orthorexia nervosa (ON) and muscle dysmorphia (MD) by email recruitment and online completion (web-based survey—WBS) and by in person recruitment and paper-and-pencil completion (paper-based survey—PBS), gives different results.


During 2 consecutive academic years, a self-reported questionnaire consisting of questions about personal characteristics and three tests for the evaluation of ON (ORTO-15), MD (MDDI-ITA), and EDs (EAT-26) were administered to two groups of undergraduates, respectively, as a WBS and a PBS.


The WBS response rate was 6.7% (N = 137), and the PBS response rate was 86.5% (N = 372). The WBS group showed a statistically significant higher prevalence of students with eating disordered behaviours (21.2% vs 5.4%) and registered a higher mean score on the EAT-26 test (13.5 ± 11.1 vs 6.0 ± 8.0); no differences between the two groups emerged for ON and MD prevalence and test scores. Moreover, in the WBS group, the number of students with one or more tests with test scores above the cut-off values was significantly higher (46.0% vs 32.3%).


The choice of the approach to administer a questionnaire to assess the diffusion of EDs and related issues must take into account all the factors that can result in selection bias and that can affect the reliability of the results.

Level of evidence

Level V, descriptive cross-sectional survey.


Several instruments used for psychological and psychiatric clinical and research applications have been validated for administration via the web [1], and this way of administering questionnaires is becoming important in the research field. The preferred mode for collecting survey data in research has traditionally been the paper questionnaire; however, in recent years, this way of collecting data has been challenged [2]. The on-going COVID-19 pandemic for example has highlighted the practical value of using online tools. For the assessment of eating disorders (EDs), since 2013, five new tools have been developed and validated exclusively for online self-report administration or for both online and pencil-and-paper administration [3]. EDs are mental disorders described in the Diagnostic and Statistical Manual of the American Psychiatric Association (DSM-5). Orthorexia nervosa (ON) is not currently recognized as a mental disorder, while muscle dysmorphia (MD) is classified in the DSM-5 as a subtype of body dysmorphic disorder. ON and MD are considered close to EDs [4,5,6,7,8,9]. The term orthorexia nervosa, literally meaning “proper appetite”, was first coined by Bratman in 1997 [10] to describe an excessive fixation on healthy eating, often associated with significant dietary restrictions and consequent life-threatening medical conditions related to malnutrition, disrupted social life and social isolation. Muscle dysmorphia was first identified by Pope et al. [11] in a group of bodybuilders and refers to individuals preoccupied with their appearance and concerned about not being sufficiently large and muscular, with a life consumed by activities aimed at increasing muscularity, such as weightlifting, dieting and using drugs [12, 13].

The prevalence of ON and MD has been assessed using self-reported questionnaires as screening tools [14,15,16,17], administered mostly as paper-and-pencil questionnaires after an in person recruitment [18,19,20,21,22,23,24,25]. More recently, questionnaires have been administered as well online, after a web-based recruitment for example via email or through an advertisement on a website [5, 26,27,28,29,30,31]. Comparability of the reliability of web-based and paper questionnaires has been supported in some cases [32,33,34]. It is widely accepted that web-based questionnaires offer advantages, which include more complete data [35], faster return [36, 37], and lower costs [38]. Two main disadvantages have been identified: (1) the relatively high nonresponse rate compared with that from traditional methods and (2) concerns regarding the reliability and validity of the data obtained [39, 40]. Furthermore, when an online test is merely an adaptation of a traditional offline instrument, evidence that the offline version has satisfactory psychometric properties is not sufficient to allow one to assume they will apply to the online version as well [41]. It is important to understand the validity of these measures and why it is necessary to know the accuracy of web vs paper-and-pencil questionnaires. For instance, since ON and MD are not currently considered disorders in the DSM-5, it is important to accurately measure ON and MD to assess their prevalence, risk factors, and correlates to examine whether to consider them as clinical disorders or to better understand them as disorders since there is growing awareness that this is how EDs present for many people, especially men.

In the present study, we compare the results of two surveys made to assess the diffusion of ON, MD and ED traits in university students, where two different approaches of administering a questionnaire were used: by email recruitment and online completion (web-based survey—WBS) and by in person recruitment and paper-and-pencil completion (paper-based survey—PBS). The two surveys were conducted at the University of Turin, Italy, during two different and consecutive academic years, enrolling the students attending the first year. The aim of the study was to evaluate if the two different approaches, WBS and PBS, could influence the results of the questionnaire.


Study design and setting

Web-based and paper-based questionnaire surveys were carried out during two different and consecutive academic years, respectively, 2013–2014 and 2014–2015, at the University of Turin.

For the web-based questionnaire survey (WBS), an online questionnaire was developed using the Lime Survey TM (Fa. Carsten Schmitz/Germany). An email invitation (and two reminders), including a link to the website to participate anonymously, were sent to the institutional email addresses of the students. The survey could be completed on all type of devices (computer, smartphone, tablet). The first page of the questionnaire included the information sheet and the informed consent; the affirmative answer to the consent allowed access to the questionnaire. It was divided into four sections, each one organized on several pages based on the number and length of the questions; to move from one page to another it was necessary to click the “next” button. It was mandatory to fill in each question before moving on to the next.

For the paper-and-pencil-based questionnaire survey (PBS), participants were approached during lessons after conferring with professors and were asked to anonymously complete the questionnaire in the classroom. The questionnaire consisted of six pages: the information sheet, the informed consent and the four sections of the questionnaire, one per page.

In both academic years, the survey was presented as an investigation among university students about nutrition habits, approach towards physical activity and proper body aspect. This research was reviewed and approved by the Bioethical Committee of the University of Turin on 01/29/2014.


The participants were students attending first year course in medicine, dietetics, physiotherapy, exercise and sport science and business administration, in two different and consecutive academic years. To participate in the survey, it was necessary to give informed consent after taking note of the informative paper.

Students were not compensated for the participation in the study. As an incentive, a personal code was given to each participant: it was included in the email invitation for the WBS and in the informative sheet and in each questionnaire for the PBS. At the end of the study, the codes of the participants with risk traits have been reported on a web page with the indication of the contacts of professionals for supporting students resulted with traits of ON, MD and EDs; in this way, students were able to know their results anonymously and, if desired, to get in touch with the professionals.


The questionnaire comprised four sections: (I) questions about personal characteristics and habits as sex, age, weight, height, hours and type of physical exercise, supplements and medicines use, and dieting, (II) the ORTO-15 test [14], which identifies individuals with ON traits, (III) the Muscle-Dysmorphic-Disorder-Inventory Italian version (MDDI-ITA) test [16], which identifies individuals with MD traits, and (IV) the Eating Attitudes Test-26 (EAT-26) [42], which identifies individuals with EDs traits.

ORTO-15 test

The ORTO-15 test was validated for the Italian population by Donini and colleagues [14]; it is composed of 15 items (for instance: “In the past 3 months, have you felt troubled by the thought of food?”, “Does the thought of food worry you for more than 3 h a day?”) using a four-point Likert scale (always, often, sometimes, never) and participants has to check one answer per item; answers that indicate a risk of ON have a score of “1”, while the “healthier” responses receive a score of “4”. The sum of the points is the final score of the test. Donini and colleagues [14] selected two threshold values below which a diagnosis of at risk of ON could be given: < 40 and < 35, identifying the value of 40 as more predictive of ON. The authors concluded that cut-off point values could be set depending on the purpose for which the scale was used. We chose the cut-off < 35, which showed a high specificity (94.2%) and negative predictive value (91.1%). The value of Cronbach’s alpha coefficient, not reported in the ORTO-15 validation study [14], resulted 0.79 in a later study on Italian athletes [43].


MDDI-ITA is a test for the presence of risk of MD. It was validated in the Italian language by Santarnecchi and Dettore [16]; the original English version was developed by Hildebrandt and colleagues [15]. It is composed of 13 items (for example: “I think my legs are too thin”, “I hate my body” or “I feel like I have too much body fat”) rated on a 5-point Likert-type scale (never, rarely, sometimes, often, always) ranging from point “1” for “never” to “5” points for always; participants has to check one answer per item. The sum of the points is the final score of the test. Cronbach’s alpha coefficient was 0.85 [16]. Currently, measurement instruments for MD have not established a defined cut-off score that allows for discrimination of clinically significant results [44]. In this study, we used a cut-off of 39 as previously adopted [18, 28, 45] on the basis of Varangis and colleagues [46], who reported a specificity of 75% and a sensitivity of 73.7%.

EAT-26 test

EAT-26 is one of the most used tests for identifying subjects with traits of EDs, and it was validated in Italy by Dotti and Lazzari [42]. It is composed of 26 items (always, usually, often, sometimes, rarely, never) and participants have to check 1 answer per item. The sum of the questions yields the total score. The test investigates three different areas of the disorder such as dieting, bulimia and food preoccupation and oral control. For instance, typical questions asked in EAT-26 test are: “I think about food with concern”, “I feel very guilty after eating” or “I feel like throwing up after eating”. The threshold value ≥ 20 identifies subjects at risk of EDs. Cronbach’s alpha coefficient was 0.86 [42].


The possible differences in the results obtained through the two different approaches of administering the questionnaire, PBS and WBS, were evaluated in terms of: characteristics of participants (sex, age, BMI, hours of physical exercise, supplements and medicines use, dieting); prevalence of traits of ON, MD and EDs; test scores of ORTO-15, MDDI-ITA and EAT-26; students with test scores above the cut-off and with the co-presence of ON, MD and ED traits; correlations between the three test scores, hours of physical activity and BMI.

Statistical analysis

All data are presented as the mean ± standard deviation or percentage, except for exercise levels that were not normally distributed and are presented as the median (interquartile range). Data were processed using SPSS software, version 25. Data analysis includes descriptive statistics, Student’s t test or a χ2 test to assess the significant differences in variables between the two groups (e.g., BMI or students with ON traits and students without ON traits), Mann–Whitney test to compare physical exercise levels, alpha reliabilities (α—Cronbach’s alpha) for each scale and Pearson-r for correlation analyses between the test scores, physical activity levels and BMI.



For the WBS, an email invitation was sent to 2047 students: 180 (8.8%) entered the web page of the questionnaire, 18 did not give their informed consent, 162 agreed to participate and 137 completed the questionnaire, with a response rate of 6.7%.

For the PBS, the questionnaire was administered to 430 students: 372 provided a complete questionnaire response, with a response rate of 86.5%. The rest of the students did not complete all the questions thus these questionnaires were not considered in the statistical analysis.

Sample characteristics and questionnaire results

The descriptive characteristics of the sample are presented in Table 1. Considering personal characteristics and habits, the group of students who filled out the paper-based questionnaire is engaged in more physical activity than the group who filled out the web-based questionnaire. Regarding the distribution of the participants in the degree courses, no difference in the distribution of medicine students between the two groups has been found, while the students of dietetics and physiotherapy and of exercise and sport science were more represented (p = 0.044) and those of business administration were less represented (p = 0.002) in the PBS group. Making a comparison between the WBS and the PBS groups, no statistically significant differences emerged by sex, BMI, supplements and medicines use or dieting. However, a statistically significant higher weekly physical activity in PBS group than in WBS was observed, probably due to the greater number of students of exercise and sport science course in the PBS group.

Table 1 Characteristics of participants

Analysing the results of the three tests that evaluated ON, MD and EDs, a statistically significant difference emerged only for EDs: the WBS group showed a higher prevalence and a higher score with the EAT-26 scale than the PBS group did, while no differences between the two student groups emerged in terms of prevalence of ON and MD traits or in terms of ORTO-15 and MDDI-ITA test scores (Tables 1, 2). As shown in Table 2, the Cronbach’s alpha values for ORTO-15 test in both groups resulted lower than that obtained in a previous study (α = 0.79) [43]; moreover, the Cohen’s d value reveals an effect due to the size differences between the WBS and PBS groups for the EAT-26 test.

Table 2 Results of the three tests (means score value ± s.d.) obtained in the WBS and PBS groups

Due to the internal consistence of ORTO-15, all the results must be interpreted with caution. As suggested by Meule and colleagues [47], we tried to reverse the score in some items (for example item 8). However, the result of Cronbach’s alpha did not change. According to the authors mentioned above, the ORTO-15 has problems in wording and scoring and a revision of the Italian version is needed.

Because of the extreme difference in number of students belonging to the different degree courses, it was impossible to make a comparison between the WBS and PBS groups of the results obtained with the three tests in the different degree courses.

Students with test scores above the cut-off and with a co-presence ON, MD and ED traits

The co-presence of a risk factor for two or three simultaneous conditions was registered in both groups. For the two groups, the analysis of the distribution of the number of students who had none, one or more test scores above the cut-off values in the three submitted test responses (ORTO-15, MDDI-ITA and EAT-26) showed a statistically significant difference: the number of students without any risk condition was higher in the group that filled out the paper-based questionnaire than that in the web-based questionnaire group, and in the same group, the number of students with one or more test scores above the cut-off values was lower than the number of web-based questionnaire students (Table 3).

Table 3 Number of students with test scores above the cut-off

Correlations among the three test scores, hours of physical activity and BMI

In the WBS and PBS groups, ON, MD and ED were correlated (Pearson correlation) with each other. In both groups of students, the ORTO-15 scores were negatively correlated with the MDDI-ITA and EAT-26 scores; a lower score on the ORTO-15 test corresponded to a greater attitude towards ON: the correlations found suggest that as the orthorexic attitudes increase, attitudes for behaviours typical of MD or EDs also increase. MDDI-ITA scores were correlated with EAT-26 scores and with hours of physical activity. Moreover, only in the PBS group, MDDI-ITA was also correlated with BMI, and the EAT-26 was negatively correlated with hours of physical activity (Table 4).

Table 4 Correlations among test scores, BMI and physical activity


In this study, we assessed the comparability of the results obtained from two surveys carried out among students at the University of Turin to evaluate the prevalence of ON, MD and ED traits using different approaches of administering the questionnaire: by email recruitment and online completion (web-based survey—WBS) and by in person recruitment and paper-and-pencil completion (paper-based survey—PBS). Surveys were carried out in two consecutive academic years and involved students attending their first year in degree courses of medicine, dietetics, physiotherapy, exercise and sport science and business administration; the number of students enrolled in university courses in the two academic years was comparable.

In the web-based questionnaire survey (WBS), 2047 students were invited to participate via email, and the response rate was very low, 6.7%, with only 162 acceptances and 137 questionnaires completed. In the paper-and-pencil-based questionnaire survey (PBS), the number of students approached and invited to participate in the survey was lower, totalling 430 students, however, the response rate was 86.5%. But all agreed to participate (100%), and the response rate was 86.5%. The shift from WBS to PBS was decided, because the response rate with WBS was very low. There were two reasons why fewer students were approached during lessons than were invited via email: (1) courses with a large number of students enrolled (i.e., business administration and exercise and sport science) divide students into more than one class, and we did not have the personnel or sufficient time to administer questionnaires in all the classes; and (2) since mandatory attendance is not required for all university courses, it is difficult to reach all students during lessons. About personal characteristics and habits, comparing the descriptive characteristics of the WBS and PBS groups, the only significant difference was a lesser amount of hours per week of physical activity in the WBS group, that is probably due to the presence of a higher number of students of Exercise and sport science course in the PBS group: the students of dietetics and physiotherapy and of exercise and sport science in fact were more represented and those of business administration were less represented.

Analysing the results of the three tests no significant differences have been revealed between WBS and PBS groups with ON and MD scales; however, the WBS group presented significantly higher scores with the EAT-26 test and a greater number of subjects with ED traits than the PBS group. Furthermore, in the WBS group, the number of students with tests with scores above the cut-off values and the number of students with one or more tests with score above the cut-off was significantly greater. While taking into account the limitation due to the different sizes of the two groups for EAT-26 test (Cohens’ d value = 0.84), these results indicate that in the WBS group, there was a major prevalence of EDs traits, and more generally, considering the number of tests with scores above the cut-off values, there were more students with traits for the conditions examined than in the PBS group. Taking into account the low response rate obtained in the WBS group, the finding of a great number of subjects with critical traits towards eating disorders and the other problems investigated in this group could be due to a selection bias of the WBS participants caused by a greater propensity of the subjects with these traits to fill out the questionnaire.

Analysing the validity of the results obtained with the three tests, another important issue is associated to the choice of ORTO-15 test for ON evaluation, which reliability measured by Cronbach’s alpha resulted low with both administration methods (WBS α = 0.305; PBS α = 0.312). The publication in which the ORTO-15 was validated dates back to 2005 and on that occasion, the value of Cronbach’s alpha was not reported. The reliability of the scale was measured in some subsequent articles, where the test was applied in groups of subjects with particular characteristics (athletes and patients with eating disorders) obtaining alpha values towards 0.79–0.81 [43, 48]. Unfortunately, the poor reliability of this test encountered in our study is in line with other studies [49], where authors tried to validate the test in other countries [47].

According to Meule and colleagues [47], ORTO-15 has problems in wording and scoring and a revision of the Italian version is needed. The review of this test is currently underway [50, 51], also in relation to the changes in the food approach and lifestyles of the Italian population that have occurred since 2005 to date. About the results of our investigation, the alpha values lead to consider the results obtained from ORTO-15 with caution.

It is known that web-based administration may yield slightly different results compared with those obtained from paper-and-pencil assessments [52, 53], and it has been documented that the mode of test administration affects the expected score distributions [41]. Moreover, web-based questionnaires have been concerned about the reliability and validity of the data obtained. Studies in various areas of health research have shown that traditional epidemiologic risk factors, such as perceived health status, anthropometry data, and smoking and alcohol use, can be collected with equal or even better reliability in web-based questionnaires than with traditional approaches [39, 54].

In our study, we attribute the differences of the test scores between online and paper-and-pencil administration to three main aspects: the low response rate in the WBS and a possible resulting selection bias, the absence of a validation of the online versions of the three tests, and a different approach of participants towards online questionnaires vs paper-and-pencil questionnaires.

Low response rates represent a major concern that threatens the quality of the web surveys [55], and self-selection is a common cause of selection bias [39]. Also, traditional modes of data-collection are facing a drop in response rates, and concerns about nonresponse bias are not only applicable to online data collection [39, 56,57,58]. Actual data on web surveys yielded, on average, a 12% points lower response rate compared with other survey modes (range from 1.4 to 82.1% response rate) [59, 60]; the response rate obtained in our study and in other studies using online assessments of ON and EDs are generally in the same range [30, 61, 62]. A rate response of 18.4% was obtained from 11,828 students at two universities in the USA after sending an email invitation to complete the EDE-Q test for assessing the prevalence of EDs [61]. Tremelling and colleagues [62] obtained a response rate of 27.4% among a sample of 2500 dietitian nutritionists invited via email to complete the ORTO-15 and EDE-Q tests in Texas, concluding that choosing to participate or not could influence results regarding the presence of ON traits among respondents [62]. Dell’Osso and colleagues [30] sent an email invitation to the whole student population at the University of Pisa, Italy, (51,609 subjects) to fill out the ORTO-15, and the response rate was 4.13%, a factor that reduced the generalizability of the study results according to the authors. A higher rate was obtained by Parra-Fernandez and colleagues [31]: on 640 university students who were asked to complete an online questionnaire through the JotForm platform, they had a response rate of 70.28%.

Among the factors that can influence the response rate in web surveys there are the sponsors, the topic, the length of time required to complete the survey, the presentation of the questionnaire, the contact delivery modes, the use of pre-notifications and the presence of incentives [55]. According to a review of web surveys [55], several meta-analyses have shown that the salience of a topic is one of the most important factors that influences the response rates to both mail and web surveys [63,64,65]. When the topic is of high salience (i.e., the topic is of high interest to some surveys), potential respondents are more likely to respond to the survey [55].

Therefore, we could hypothesize that in our study, the topic of the survey presented through the WBS attracted a high number of subjects interested in issues related to nutrition, body aspect and physical activity. In contrast, in the PBS, all students approached in class agreed to participate, reducing the self-selection bias of the sample.

To our knowledge, there is no validation of the online version of MDDI-ITA or of the EAT-26 test. The validity of an online version of the ORTO-15 translated into Portuguese was tested among a sample of Brazilian dieticians, but no evidence was found of its validity and reliability with the initial psychometric evaluation performed [66]. Some studies used online adaptations of ORTO-15 [26, 27, 30, 62, 67, 68], MDDI-ITA [28] or its original English version [69] and EAT-26 [5, 69,70,71]. While most of the evidence to date indicates that online adaptations of offline tests usually address the expected constructs, there have been sufficient indications of (usually small) differences (e.g., in factor structure, score distributions) to advocate caution, especially in instances where test use has real implications for people’s well-being [41]. According to Buchanan, when an online test is an adaptation of a traditional offline instrument, evidence that the offline version has satisfactory psychometric properties is not sufficient to allow one to assume they will apply to the online version as well [41]. In our study, the use of tools not even validated for psychometric properties for online administration can be a limitation in the results obtained.

A strong candidate for explaining the reasons behind the differences in the scores between online and paper-and-pencil surveys is increased self-disclosure [1]. There is compelling evidence that people may disclose more about themselves when communicating via computers than via face-to-face interactions [72], a phenomenon that appears to extend to internet-mediated communication [73]. This has actually been one of the possible advantages suggested for online clinical work and has also led to the suggestion that online psychological questionnaires will actually give a better picture of the individual’s real personality than traditional measures would [1]. Electronic administration of questionnaires can affect the responses given to self-administered survey questionnaires through direct influence on the respondents [74]. For example, concerns about privacy, anonymity and confidentiality might influence the accuracy of the answers to certain items, and social and cultural beliefs can influence the acceptability of the response [74, 75]. In the PBS, the students completed the questionnaire in the classroom; this aspect may have affected the self-disclosure, as students were not alone during the completion but with their classmates.

A correlation among the three test scores was identified in both the WBS and PBS groups: the increase in attitudes towards one of the conditions examined also implies an increase in attitudes towards the other two conditions. This is in line with the literature: a correlation between ON and ED traits emerged in some studies [6, 67, 76, 77] as well as a correlation between MD and EDs [78,79,80]. This result is also in line with the studies in which a proximity of features between ON and MD with EDs is considered [4,5,6,7,8,9, 25].

The use of electronic self-administered survey questionnaires has become common in several research areas [81]. In situations such as the current COVID-19 pandemic, the importance of using online tools emerges more than ever. It is, therefore, essential that these tools are valid and representative and it is important to consider the impact that changing the mode of delivery can have on the responses collected [74]. In this study, the same questionnaire for evaluating the prevalence of ON, MD and ED traits administered in 2 consecutive years to an analogous group of undergraduates online and via paper-and-pencil gave some different results: in comparison with the PBS group students who filled the online questionnaire had a higher prevalence of traits for EDs and a higher number of subjects with traits for one or more of the three conditions examined. These differences could be due to an effective distinction between the two groups, but the size differences between the two groups and the different approach for the questionnaire administration can also play a significant role. It is important to address correctly online surveys, preferably requiring instruments specifically validated for that use.

This investigation has some limitation. The first is related to the different numbers of students participating at the surveys in the two groups (n = 137 in the WBS group; n = 372 in the PBS group) that could have affected some results, as for EAT-26 (Cohen’s d value = 0.84) and the particularly low response rate obtained with the WBS (6.7%) could be another problem concerning the interpretation of the results obtained with the three tests. Another limitation is about the social desirability responding (the tendency to reply to questionnaire giving a more favorable image of him/her-self [82]. About the scales, as discussed above, the ORTO-15 needs a revision in wording and scoring: the internal consistence of this scale highlights the limits of the Italian version of this measure. Moreover, the two study cohorts used to obtain online and paper–pencil show differences regarding type of study and physical activity: these variables can be included as covariates in further analyses. Finally, for the limited information we gathered on this subject, we could not control the effect of socio-economic variables on the scores obtained from paper–pencil and online assessments, and therefore, correct the results for the background characteristics of participants.

Despite these limitations, results from this investigation could help scholars to choose the different type of administration of ORTO-15, MDDI-ITA and EAT-26 questionnaire: findings show that WBS, in particular, permits a larger participation than PBS.

What is already know on the subject?

Online surveys to assess the diffusion of eating disorders, both classified and emerging as orthorexia nervosa and muscle dysmorphia have been widely used in recent years, however, the questionnaires used are not generally validated for online administration. The results of online surveys may be affected by bias due for example to low response rates, to a self-selection linked to the salience of a topic, the sponsors, the length of time required to complete the survey, the presentation of the questionnaire, the contact delivery modes, the use of pre-notifications and the presence of incentives. Furthermore, in online surveys, subjects often have greater self-disclosure. The results of the web-based surveys must take into account all these aspects to be considered valid and reliable.

What your study adds?

This study, for the first time to our knowledge, compares the results obtained with the online and paper administration of questionnaires for the evaluation of the diffusion of EDs, ON and MD on analogous groups of university students. Differences between the groups have been identified. The results highlight the need for an adequate design of web-based surveys and the importance of using validated questionnaires for this type of administration.

Data availability

The dataset generated and analysed during the current study are available from the corresponding author on reasonable request.


  1. 1.

    Buchanan T (2003) Internet-based questionnaire assessment: appropriate use in clinical contexts. Cogn Behav Ther.

    Article  PubMed  Google Scholar 

  2. 2.

    Ebert JF, Huibers L, Christensen B, Christensen MB (2018) Paper- or web-based questionnaire invitations as a method for data collection: cross-sectional comparative study of differences in response rate, completeness of data, and financial cost corresponding author. J Med Internet Res 20:1–12.

    Article  Google Scholar 

  3. 3.

    Forbush KT, Gould SR, Chapa DAN et al (2017) New horizons in measurement: a review of novel and innovative approaches to eating-disorder assessment. Curr Psychiatry Rep.

    Article  PubMed  Google Scholar 

  4. 4.

    Dell’Osso L, Abelli M, Carpita B et al (2016) Historical evolution of the concept of anorexia nervosa and relationships with orthorexia nervosa, autism, and obsessive-compulsive spectrum. Neuropsychiatr Dis Treat 12:1651–1660.

    Article  PubMed  PubMed Central  Google Scholar 

  5. 5.

    Bundros J, Clifford D, Silliman K, Neyman Morris M (2016) Prevalence of Orthorexia nervosa among college students based on Bratman’s test and associated tendencies. Appetite 101:86–94.

    Article  PubMed  Google Scholar 

  6. 6.

    Sanlier N, Yassibas E, Bilici S et al (2016) Does the rise in eating disorders lead to increasing risk of orthorexia nervosa? Correlations with gender, education, and body mass index. Ecol Food Nutr 55:266–278.

    Article  PubMed  Google Scholar 

  7. 7.

    Barthels F, Meyer F, Huber T, Pietrowsky R (2017) Orthorexic eating behaviour as a coping strategy in patients with anorexia nervosa. Eat Weight Disord 22:269–276.

    Article  PubMed  Google Scholar 

  8. 8.

    Mitchison D, Mond J (2015) Epidemiology of eating disorders, eating disordered behaviour, and body image disturbance in males: a narrative review. J Eat Disord 3:1–9.

    Article  Google Scholar 

  9. 9.

    Murray SB, Nagata JM, Griffiths S et al (2017) The enigma of male eating disorders: a critical review and synthesis. Clin Psychol Rev 57:1–11.

    Article  PubMed  Google Scholar 

  10. 10.

    Bratman S (1997) Health food junkie. Yoga J 136:42–50

    Google Scholar 

  11. 11.

    Pope HG, Katz DL, Hudson JI (1993) Anorexia nervosa and “reverse anorexia” among 108 male bodybuilders. Compr Psychiatry 34:406–409.

    Article  PubMed  Google Scholar 

  12. 12.

    Tod D, Edwards C, Cranswick I (2016) Muscle dysmorphia: current insights. Psychol Res Behav Manag 9:179–188.

    Article  PubMed  PubMed Central  Google Scholar 

  13. 13.

    Pope HG, Gruber AJ, Choi P et al (1997) Muscle dysmorphia: an underrecognized form of body dysmorphic disorder. Psychosomatics 38:548–557.

    Article  PubMed  Google Scholar 

  14. 14.

    Donini LM, Marsili D, Graziani MP et al (2005) Orthorexia nervosa: validation of a diagnosis questionnaire. Eat Weight Disord Stud Anorex Bulim Obes 10:e28–e32.

    CAS  Article  Google Scholar 

  15. 15.

    Hildebrandt T, Langenbucher J, Schlundt DG (2004) Muscularity concerns among men: development of attitudinal and perceptual measures. Body Image 1:169–181.

    Article  PubMed  Google Scholar 

  16. 16.

    Santarnecchi E, Dèttore D (2012) Muscle dysmorphia in different degrees of bodybuilding activities: validation of the Italian version of muscle dysmorphia disorder inventory and bodybuilder image grid. Body Image 9:396–403.

    Article  PubMed  Google Scholar 

  17. 17.

    Olivardia R, Pope J, Hudson JI (2000) Muscle dysmorphia in male weightlifters: a case-control study. Am J Psychiatry 157:1291–1296.

    CAS  Article  PubMed  Google Scholar 

  18. 18.

    Bo S, Zoccali R, Ponzo V et al (2014) University courses, eating problems and muscle dysmorphia: are there any associations? J Transl Med 12:221.

    Article  PubMed  PubMed Central  Google Scholar 

  19. 19.

    Baǧci Bosi AT, Çamur D, Güler Ç (2007) Prevalence of orthorexia nervosa in resident medical doctors in the faculty of medicine (Ankara, Turkey). Appetite 49:661–666.

    Article  PubMed  Google Scholar 

  20. 20.

    Catherine Walker D, Anderson DA, Hildebrandt T (2009) Body checking behaviors in men. Body Image 6:164–170.

    Article  PubMed  Google Scholar 

  21. 21.

    Murray SB, Rieger E, Hildebrandt T et al (2012) A comparison of eating, exercise, shape, and weight related symptomatology in males with muscle dysmorphia and anorexia nervosa. Body Image 9:193–200.

    Article  PubMed  Google Scholar 

  22. 22.

    Malmborg J, Bremander A, Olsson MC, Bergman S (2017) Health status, physical activity, and orthorexia nervosa: a comparison between exercise science students and business students. Appetite 109:137–143.

    Article  PubMed  Google Scholar 

  23. 23.

    Brytek-Matera A, Fonte ML, Poggiogalle E et al (2017) Orthorexia nervosa: relationship with obsessive-compulsive symptoms, disordered eating patterns and body uneasiness among Italian university students. Eat Weight Disord 22:609–617.

    Article  PubMed  Google Scholar 

  24. 24.

    Compte EJ, Sepulveda AR, Torrente F (2015) A two-stage epidemiological study of eating disorders and muscle dysmorphia in male university students in Buenos Aires. Int J Eat Disord 48:1092–1101.

    Article  PubMed  Google Scholar 

  25. 25.

    Gorrasi ISR, Bonetta S, Roppolo M et al (2020) Traits of orthorexia nervosa and muscle dysmorphia in Italian university students: a multicentre study. Eat Weight Disord 25:1413–1423.

    Article  PubMed  Google Scholar 

  26. 26.

    Dell’Osso L, Abelli M, Carpita B et al (2016) Orthorexia nervosa in a sample of Italian university population. Riv Psichiatr 51:190–196.

    Article  PubMed  Google Scholar 

  27. 27.

    Dunn TM, Gibbs J, Whitney N, Starosta A (2017) Prevalence of orthorexia nervosa is less than 1 %: data from a US sample. Eat Weight Disord 22:185–192.

    Article  PubMed  Google Scholar 

  28. 28.

    Longobardi C, Prino LE, Fabris MA, Settanni M (2017) Muscle dysmorphia and psychopathology: Findings from an Italian sample of male bodybuilders. Psychiatry Res 256:231–236.

    Article  PubMed  Google Scholar 

  29. 29.

    Oberle CD, Samaghabadi RO, Hughes EM (2017) Orthorexia nervosa: assessment and correlates with gender, BMI, and personality. Appetite 108:303–310.

    Article  PubMed  Google Scholar 

  30. 30.

    Dell’Osso L, Carpita B, Muti D et al (2018) Prevalence and characteristics of orthorexia nervosa in a sample of university students in Italy. Eat Weight Disord 23:55–65.

    Article  PubMed  Google Scholar 

  31. 31.

    Parra-Fernández ML, Onieva-Zafra MD, Fernández-Martínez E et al (2019) Assessing the prevalence of orthorexia nervosa in a sample of university students using two different self-report measures. Int J Environ Res Public Health.

    Article  PubMed  PubMed Central  Google Scholar 

  32. 32.

    Ritter P, Lorig K, Laurent D, Matthews K (2004) Internet versus mailed questionnaires: a randomized comparison. J Med Internet Res.

    Article  PubMed  PubMed Central  Google Scholar 

  33. 33.

    Whitehead L (2011) Methodological issues in internet-mediated research: a randomized comparison of internet versus mailed questionnaires. J Med Internet Res.

    Article  PubMed  PubMed Central  Google Scholar 

  34. 34.

    Duracinsky M, Lalanne C, Goujard C et al (2014) Electronic versus paper-based assessment of health-related quality of life specific to HIV disease: reliability study of the PROQOL-HIV questionnaire. J Med Internet Res.

    Article  PubMed  PubMed Central  Google Scholar 

  35. 35.

    Bech M, Kristensen MB (2009) Differential response rates in postal and web-based surveys among older respondents. Surv Res Methods.

    Article  Google Scholar 

  36. 36.

    Akl EA, Maroun N, Klocke RA et al (2005) Electronic mail was not better than postal mail for surveying residents and faculty. J Clin Epidemiol.

    Article  PubMed  Google Scholar 

  37. 37.

    Kroth PJ, McPherson L, Leverence R et al (2009) Combining Web-based and mail surveys improves response rates: a PBRN study from PRIME Net. Ann Fam Med.

    Article  PubMed  PubMed Central  Google Scholar 

  38. 38.

    Scott A, Jeon SH, Joyce CM et al (2011) A randomised trial and economic evaluation of the effect of response mode on response rate, response bias, and item non-response in a survey of doctors. BMC Med Res Methodol 11:1–12

    Article  Google Scholar 

  39. 39.

    Van Gelder MMHJ, Bretveld RW, Roeleveld N (2010) Practice of epidemiology web-based questionnaires: the future in epidemiology? Am J Epidemiol 172:1292–1298.

    Article  PubMed  Google Scholar 

  40. 40.

    Best SJ, Krueger B, Hubbard C, Smith A (2001) An assessment of the generalizability of internet surveys. Soc Sci Comput Rev 19:131–145

    Article  Google Scholar 

  41. 41.

    Buchanan T, Buchanan T (2010) Internet-based questionnaire assessment: appropriate use in clinical contexts internet-based questionnaire assessment: appropriate use in clinical contexts. Cogn Behav Ther.

    Article  Google Scholar 

  42. 42.

    Dotti A, Lazzari R (1998) Validation and reliability of the Italian EAT-26. Eat Weight Disord Stud Anorex Bulim Obes 3:188–194.

    CAS  Article  Google Scholar 

  43. 43.

    Segura-García C, Papaianni MC, Caglioti F et al (2003) Orthorexia nervosa: a frequent eating disordered behavior in athletes. Eat Weight Disord Stud Anorex Bulim Obes 17:e226–e233.

    Article  Google Scholar 

  44. 44.

    Suffolk MT, Dovey TM, Goodwin H, Meyer C (2013) Muscle dysmorphia: methodological issues, implications for research. Eat Disord 21:437–457.

    Article  PubMed  Google Scholar 

  45. 45.

    Devrim A, Bilgic P, Hongu N (2018) Is there any relationship between body image perception, eating disorders, and muscle dysmorphic disorders in male bodybuilders? Am J Mens Health.

    Article  PubMed  PubMed Central  Google Scholar 

  46. 46.

    Varangis E, Folberth W, Hildebrandt T, Langenbucher J (2012) Confirmatory factor analysis for the Muscle Dysmorphic Disorder Inventory among male appearance and performance enhancing drug users. In: Poster Present. Int. Conf. Eat. Disord. Austin, TX

  47. 47.

    Meule A, Holzapfel C, Brandl B et al (2020) Measuring orthorexia nervosa: a comparison of four self-report questionnaires. Appetite.

    Article  PubMed  Google Scholar 

  48. 48.

    Segura-Garcia C, Ramacciotti C, Rania M et al (2015) The prevalence of orthorexia nervosa among eating disorder patients after treatment. Eat Weight Disord 20:161–166.

    Article  PubMed  Google Scholar 

  49. 49.

    Valente M, Syurina EV, Donini LM (2019) Shedding light upon various tools to assess orthorexia nervosa: a critical literature review with a systematic search. Eat Weight Disord 24:671–682.

    Article  PubMed  PubMed Central  Google Scholar 

  50. 50.

    Cena H, Barthels F, Cuzzolaro M et al (2019) Definition and diagnostic criteria for orthorexia nervosa: a narrative review of the literature. Springer International Publishing

    Google Scholar 

  51. 51.

    Cuzzolaro M, Donini LM (2016) Orthorexia nervosa by proxy? Eat Weight Disord 21:549–551.

    Article  PubMed  Google Scholar 

  52. 52.

    Donker T, van Straten A, Marks I, Cuijpers P (2010) Brief self-rated screening for depression on the Internet. J Affect Disord.

    Article  PubMed  Google Scholar 

  53. 53.

    Coles ME, Cook LM, Blake TR (2007) Assessing obsessive compulsive symptoms and cognitions on the internet: evidence for the comparability of paper and internet administration. Behav Res Ther.

    Article  PubMed  Google Scholar 

  54. 54.

    Al-Asadi AM, Klein B, Meyer D (2015) Multiple comorbidities of 21 psychological disorders and relationships with psychosocial variables: a study of the online assessment and diagnostic system within a web-based population. J Med Internet Res.

    Article  PubMed  PubMed Central  Google Scholar 

  55. 55.

    Fan W, Yan Z (2010) Computers in human behavior factors affecting response rates of the web survey: a systematic review. Comput Hum Behav 26:132–139.

    Article  Google Scholar 

  56. 56.

    Nohr EA, Frydenberg M, Henriksen TB, Olsen J (2006) Does low participation in cohort studies induce bias? Epidemiology.

    Article  PubMed  Google Scholar 

  57. 57.

    Laaksonen M, Aittomäki A, Lallukka T et al (2008) Register-based study among employees showed small nonparticipation bias in health surveys and check-ups. J Clin Epidemiol.

    Article  PubMed  Google Scholar 

  58. 58.

    Hohwü L, Lyshol H, Gissler M et al (2013) Web-based versus traditional paper questionnaires: a mixed-mode survey with a Nordic perspective corresponding author. J Med Internet Res 15:1–11.

    Article  Google Scholar 

  59. 59.

    Braekman E, Charafeddine R, Demarest S et al (2020) Comparing web-based versus face-to-face and paper-and-pencil questionnaire data collected through two Belgian health surveys. Int J Public Health 65:5–16.

    Article  PubMed  Google Scholar 

  60. 60.

    Daikeler J, Bošnjak M, Lozar Manfreda K (2020) Web versus other survey modes: an updated and extended meta-analysis comparing response rates. J Surv Stat Methodol 8:513–539.

    Article  Google Scholar 

  61. 61.

    Ketchen S, Jones JM, Taylor CB et al (2017) Understanding and promoting treatment-seeking for eating disorders and body image concerns on college campuses through online screening, prevention and intervention. Eat Behav 25:68–73.

    Article  Google Scholar 

  62. 62.

    Tremelling K, Sandon L, Vega GL, McAdams CJ (2017) Orthorexia nervosa and eating disorder symptoms in registered dietitian nutritionists in the United States. J Acad Nutr Diet 117:1612–1617.

    Article  PubMed  PubMed Central  Google Scholar 

  63. 63.

    Cook C, Heath F, Thompson RL (2000) A meta-analysis of response rates in web- or internet-based surveys. Educ Psychol Meas.

    Article  Google Scholar 

  64. 64.

    Edwards P, Roberts I, Clarke M et al (2002) Increasing response rates to postal questionnaires: systematic review. Br Med J 324:1183

    Article  Google Scholar 

  65. 65.

    Yammarino FJ, Skinner SJ, Childers TL (1991) Understanding mail survey response behavior a meta-analysis. Public Opin Q.

    Article  Google Scholar 

  66. 66.

    Dos M, Alvarenga S, Martins M (2012) Orthorexia nervosa behavior in a by the Portuguese version of ORTO-15. Eat Weight Disord Stud Anorex Bulim Obes 17:e29-235.

    Article  Google Scholar 

  67. 67.

    Asil E, Sürücüoğlu MS (2015) Orthorexia nervosa in Turkish dietitians. Ecol Food Nutr 54:303–313.

    Article  PubMed  Google Scholar 

  68. 68.

    Barnes MA, Caltabiano ML (2017) The interrelationship between orthorexia nervosa, perfectionism, body image and attachment style. Eat Weight Disord 22:177–184.

    Article  PubMed  Google Scholar 

  69. 69.

    Mitchell L, Murray SB, Hoon M et al (2017) Correlates of muscle dysmorphia symptomatology in natural bodybuilders: distinguishing factors in the pursuit of hyper-muscularity. Body Image 22:1–5.

    Article  PubMed  Google Scholar 

  70. 70.

    Satherley R, Howard R, Higgs S (2016) The prevalence and predictors of disordered eating in women with coeliac disease. Appetite 107:260–267.

    Article  PubMed  Google Scholar 

  71. 71.

    Darcy AM, Dooley B (2007) A clinical profile of participants in an online support group. Eur Eat Disord Rev 195:185–195.

    Article  Google Scholar 

  72. 72.

    Weisband S, Kiesler S (1996) Self disclosure on computer forms: meta-analysis and implications. In: Conference on human factors in computing systems - proceedings

  73. 73.

    Joinson A (1999) Social desirability, anonymity, and internet-based questionnaires. Behav Res Methods Instrum Comput.

    Article  PubMed  Google Scholar 

  74. 74.

    Belisario JSM, Jamsek J, Huckvale K, O'Donoghue J, Morrison CP, Car J (2015) Comparison of self-administered survey questionnaire responses collected using mobile apps versus other methods. Cochrane Database Syst Rev.

    Article  Google Scholar 

  75. 75.

    Cheng KG, Ernesto F, Ovalle-Bahamón RE, Truong KN (2011) Barriers to acceptance of personal digital assistants for HIV/AIDS data collection in Angola. Int J Med Inform.

    Article  PubMed  PubMed Central  Google Scholar 

  76. 76.

    Agopyan A, Kenger EB, Kermen S et al (2019) The relationship between orthorexia nervosa and body composition in female students of the nutrition and dietetics department. Eat Weight Disord 24:257–266.

    Article  PubMed  Google Scholar 

  77. 77.

    Reynolds R (2018) Is the prevalence of orthorexia nervosa in an Australian university population 6. 5 %? Eat Weight Disord Stud Anorex Bulim Obes 23:453–458.

    Article  Google Scholar 

  78. 78.

    Lamanna J, Grieve FG, Pitt Derryberry W et al (2010) Antecedents of eating disorders and muscle dysmorphia in a non-clinical sample. Eat Weight Disord.

    Article  PubMed  Google Scholar 

  79. 79.

    Pope HG Jr, Phillips KA, Olivardia R (2000) The Adonis complex: the secret crisis of male body image obsession. Free Press, New York

    Google Scholar 

  80. 80.

    Hildebrandt T, Schlundt D, Langenbucher J, Chung T (2006) Presence of muscle dysmorphia symptomology among male weightlifters. Compr Psychiatry 47:127–135.

    Article  PubMed  Google Scholar 

  81. 81.

    Lane SJ, Heddle NM, Arnold E, Walker I (2006) A review of randomized controlled trials comparing the effectiveness of hand held computers with paper methods for data collection. BMC Med Inform Decis Mak 6:1–10

    Article  Google Scholar 

  82. 82.

    van de Mortel TF (2008) Faking it: social desirability response bias in selfreport research. Aust J Adv Nurs 25:40–48

    Google Scholar 

Download references


Open access funding provided by Università degli Studi di Torino within the CRUI-CARE Agreement. This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Author information




Conceptualization: ISRG, GAD, SB, EC; methodology: ISRG, GAD, SB, EC, AT; formal analysis and investigation: ISRG, RD, MR, CF, MG, DAM; writing—original draft preparation: ISRG, RD, EC, CF, MG; writing—review and editing: ISRG, GAD, SB, EC, AT, CF, MG, DAM; supervision: GG, EC.

Corresponding author

Correspondence to Ilaria Silvia Rossella Gorrasi.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

All procedures performed in this study were in accordance with the ethical standards of the national ethics committee and with the 1964 Declaration of Helsinki. The study was approved by the bioethical committees of University of Turin.

Informed consent

Informed consent was obtained from all participants included in the study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 271 kb)

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Gorrasi, I.S.R., Ferraris, C., Degan, R. et al. Use of online and paper-and-pencil questionnaires to assess the distribution of orthorexia nervosa, muscle dysmorphia and eating disorders among university students: can different approaches lead to different results?. Eat Weight Disord (2021).

Download citation


  • Orthorexia nervosa
  • Muscle dysmorphia
  • Eating disorders
  • Web-based survey
  • Paper and pencil survey
  • Questionnaire