Investigating the structure of ORTO-15: a meta-analytical simulation study

Missbach et al. (Appetite 108:521–524, 10.1016/j.appet.2016.07.010, 2016) argued that there is a critical need to develop new tools assessing orthorexia nervosa (ON), as the existing measure (i.e., ORTO-15; Donini, Eat Weight Disord 10:28–32, 10.1007/BF03327537, 2005) is an unvalidated measure, which fails to adequately assess the prevalence rate of ON. We believe that ignoring past data from ORTO-15 and going in the “baby with the bath water” direction will not catalyse but inhibit ON research. Using data from the review of the psychometric studies analysing the structure of ORTO-15 provided in Missbach et al. (2016), we selected six items, which were present in each study, and estimated effect sizes for the factor loadings. The effect sizes were used in a Monte Carlo simulation study with N = 100, 500, and 1000 to test whether the analysed model is valid. The obtained results confirmed that the six-item version of ORTO-15 is a valid and reliable measure of ON. Although new measures of ON are needed, the past data also provide valuable insight into a better understanding of ON.


Introduction
Orthorexia nervosa (ON) is defined as pathological fixation with healthy eating which shares some common characteristics with eating disorders and obsessive-compulsive disorder [1]. Dunn and Bratman [2] proposed diagnostic criteria defining ON as obsessive focus on "healthy" eating, for which compulsive behaviour and mental preoccupation lead to clinical impairments. Although the construct of ON is well defined [1,2] and the research in the field is gaining interest (as expressed in the 1180 relevant results in the Google Scholar database for the search "orthorexia nervosa" during the last 5 years), the measurement of ON is open to doubt [3].
The most popular measure of ON, the ORTO-15 [4], which is a 15-item self-report measure on which respondents answer using a 4-point Likert-type scale, was developed as a test for assessment of ON. Using receiver operating characteristic curve analysis, Donini et al. [4] suggested that a threshold of 40 points is optimal in terms of sensitivity and specificity. Whereas the prevalence rate of ON in Donini et al. [4] reached 6.9%, Dunn and Bratman [2] demonstrated, in a review of the literature, that it ranges from 41.9 to 88.7%, which calls into question whether the ORTO-15 is a tool suitable for the assessment of ON. Another weakness of the ORTO-15 [4] is the fact that during its development, the underlying structure of the measure was not assessed and only a three-factor solution was assumed. Five studies investigated the structure of the ORTO-15, and their results were ambiguous, some of them suggesting the presence of one factor [5,6], some two [7], and some three [8,9]; moreover, in each study the authors decided to remove some of the items that seemed not to tap the assumed construct, leaving only six items that were in common to all studies.
Missbach et al. [3] argued that the broad range in prevalence rates is rooted in the assessment tool, and they discouraged using the ORTO-15 as a measure to assess ON. While such results clearly provide evidence that diagnosis made on the basis of self-report score from the ORTO-15 is troublesome, we argue that this should not The article is part of the Topical Collection on Orthorexia Nervosa.
* Radosław Rogoza r.rogoza@uksw.edu.pl 1 Cardinal Stefan Wyszyński University, Wóycickiego 1/3, 01-938 Warsaw, Poland nullify the score's usage. This is because we believe that ORTO-15 is not a good measure for categorical diagnosis, but it might be good in a dimensional assessment of ON. Such a switch corroborates recent work, for example, that of Clarkin and Livesley [10], which has claimed that all psychological traits, and thus including ON, are in fact dimensional. Missbach et al. [3] also questioned the validity of the ORTO-15, and within the current study we aim to provide evidence from a simulation study that after certain improvements, ORTO-15 may be deemed as a valid measure.
We assume that previous studies were successful in determining which items are suboptimal in the assessment of ON and were systematically excluded. From the initial pool of the 15 items, 6 of them were selected in each study [5][6][7][8][9]; therefore, to highlight the distinctiveness from all previous versions, we label it as ORTO-6. In Table 1, we present the standardized factor loadings of these items originating from k = 5 studies with total N = 3425, and on the basis of the meta-analysis we provide estimated effect sizes with 95% confidence intervals.

Simulation
On the basis of the estimates from the meta-analysis, nine Monte Carlo simulations were run, each with 1000 replications. The values of items' residual variances were assumed to be tau equivalent and equalled a quarter. We calculated a separate simulation for the exact, lower-bound, and higherbound estimates under three conditions regarding the sample size: with 100, 500, and 1000 observations. The data generated from these simulations were the basis for the assessment of the structural validity of the measure. All of the data used for analyses as well as the syntax used for data generation are freely available at https ://osf.io/qgs6u . Table 2 presents model fit indices and reliability estimates for the measurement model of ORTO-6 under different conditions.

3
The results revealed that all of the models, including the lower-bound model, were well fitted to the data and reliable in their measurement. This implies that the ORTO-6 is a structurally valid and reliable measure.

Conclusion
Missbach et al. [3] stated that previous results of studies obtained from existing ON measures of low quality should be treated with caution. On the basis of conducted simulations, we disagree with such a categorical statement. Indeed, we agree that ORTO-15 (or ORTO-6) is not a valid tool to diagnose ON and to study its prevalence; however, we argue that the tool could be successfully used for the dimensional assessment of ON. Previous research, predominantly using this measure, led to a better understanding of ON, as, for example, such studies provided evidence for its associations with eating disorders and health behaviours, its intensity in dieting styles, or gender differences [1,[4][5][6][7][8][9]. We agree that there is a need for a tool to assess ON in accordance with the diagnostics criteria [2,3]; however, we also believe that ORTO-6 might be successfully used as a general marker of orthorexic behaviours used in dimensional diagnosis [10].

Limitations
The current study provides support for using an abbreviated version of the ORTO questionnaire for the assessment of ON; however, the presented results are neither final nor definitive. Within the simulations, only those items that had not been excluded in several published studies [5][6][7][8][9] were included; however, it is still possible that the analysis carried out on all items would lead to a different item selection. Moreover, the factor loadings included in the meta-analysis were included from different measurement models, which might have influenced the obtained results. Finally, as the assumption of the tau equivalence is seldom met, the structure of the ORTO-6 should be verified by subsequent empirical studies.

Compliance with ethical standards
Conflict of interest The author declares that he has no conflict of interest.

Ethical approval
The research was conducted with respect to the ethical recommendations for psychological research.
Informed consent As the study was based on simulations and was not conducted on human participants, no informed consent was collected.
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creat iveco mmons .org/licen ses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.