Skip to main content
SpringerLink
Log in

Checking quality of sensory data via an agreement-based approach

  • Published:
Quality & Quantity Aims and scope Submit manuscript

Abstract

Sensory evaluations are adopted in many fields for measuring and comparing sensory properties of products and improving their quality. The selection of panelists able to provide precise evaluations is a crucial issue to perform reliable sensory analysis. An agreement-based approach is here suggested in order to assess the quality of sensory data in terms of both panelist repeatability and panel reproducibility. The approach has been applied to two case studies involving untrained sensory panelists and trained teaching quality assessors, respectively. The results of the case studies show that although reproducibility can be assumed moderate for both groups of raters, repeatability is generally higher for the group of trained raters.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  • Altman, D.G.: Practical Statistics for Medical Research. CRC Press, Boca Raton (1990)

    Google Scholar 

  • Bashkansky, E., Dror, S., Ravid, R., Grabov, P.: Effectiveness of a product quality classifier. Qual. Control Appl. Stat. 53(3), 291–292 (2008)

    Google Scholar 

  • Bi, J.: Agreement and reliability assessments for performance of sensory descriptive panel. J. Sens. Stud. 18(1), 61–76 (2003)

    Article  Google Scholar 

  • Brennan, R.L., Prediger, D.J.: Coefficient kappa: some uses, misuses, and alternatives. Educ. Psychol. Meas. 41(3), 687–699 (1981)

    Article  Google Scholar 

  • Brockhoff, P.B.: Statistical testing of individual differences in sensory profiling. Food Qual. Prefer. 14(5–6), 425–434 (2003)

    Article  Google Scholar 

  • Carpenter, J., Bithell, J.: Bootstrap confidence intervals: when, which, what? A practical guide for medical statisticians. Stat. Med. 19(9), 1141–1164 (2000)

    Article  Google Scholar 

  • Cicchetti, D.V., Allison, T.: A new procedure for assessing reliability of scoring eeg sleep recordings. Am. J. EEG Technol. 11(3), 101–110 (1971)

    Article  Google Scholar 

  • Cohen, J.: A coefficient of agreement for nominal scales. Educ. Psychol. Meas. 20(1), 37–46 (1960)

    Article  Google Scholar 

  • De Mast, J.: Agreement and kappa-type indices. Am. Stat. 61(2), 148–153 (2007)

    Article  Google Scholar 

  • Erdmann, T.P., De Mast, J., Warrens, M.J.: Some common errors of experimental design, interpretation and inference in agreement studies. Stat. Methods Med. Res. 24(6), 920–935 (2015)

    Article  Google Scholar 

  • Fleiss, J.L., Levin, B., Paik, M.C.: Statistical methods for rates and proportions. Wiley (2013)

  • Gadrich, T., Bashkansky, E.: Ordanova: analysis of ordinal variation. J. Stat. Plan. Inference 142(12), 3174–3188 (2012)

    Article  Google Scholar 

  • Gadrich, T., Bashkansky, E., Zitikis, R.: Assessing variation: a unifying approach for all scales of measurement. Qual. Quant. 49(3), 1145–1167 (2015)

    Article  Google Scholar 

  • Gardner, M.J., Altman, D.G.: Confidence intervals rather than P values: estimation rather than hypothesis testing. Br. Med. J. (Clin Res Ed) 292(6522), 746–750 (1986)

    Article  Google Scholar 

  • Geier, U., Büssing, A., Kruse, P., Greiner, R., Buchecker, K.: Development and application of a test for food-induced emotions. PLoS ONE 11(11), 1–17 (2016)

    Article  Google Scholar 

  • Gwet, K.L.: Handbook of Inter-Rater Reliability: The Definitive Guide to Measuring the Extent of Agreement Among Raters. Advanced Analytics, LLC, Gaithersburg (2014)

    Google Scholar 

  • Iannario, M., Manisera, M., Piccolo, D., Zuccolotto, P.: Sensory analysis in the food industry as a tool for marketing decisions. Adv. Data Anal. Classif. 6(4), 303–321 (2012)

    Article  Google Scholar 

  • International Organization for Standardization (ISO). Accuracy (Trueness and Precision) of Measurement Methods and Results Part 1: General Principles and Definitions (5725-1). Geneva: ISO (1994)

  • Kermit, M., Lengard, V.: Assessing the performance of a sensory panel–panellist monitoring and tracking. J. Chemom. 19(3), 154–161 (2005)

    Article  Google Scholar 

  • King, M.C., Hall, J., Cliff, M.A.: A comparison of methods for evaluating the performance of a trained sensory panel. J. Sens. Stud. 16(6), 567–581 (2001)

    Article  Google Scholar 

  • Landis, J.R., Koch, G.G.: The measurement of observer agreement for categorical data. Biometrics 33(1), 159–174 (1977)

    Article  Google Scholar 

  • Latreille, J., Mauger, E., Ambroisine, L., Tenenhaus, M., Vincent, M., Navarro, S., Guinot, C.: Measurement of the reliability of sensory panel performances. Food Qual. Prefer. 17(5), 369–375 (2006)

    Article  Google Scholar 

  • Lea, P., Rødbotten, M., Næs, T.: Measuring validity in sensory analysis. Food Qual. Prefer. 6(4), 321–326 (1995)

    Article  Google Scholar 

  • Ludbrook, J.: Statistical techniques for comparing measurers and methods of measurement: a critical review. Clin. Exp. Pharmacol. Physiol. 29(7), 527–536 (2002)

    Article  Google Scholar 

  • Lundahl, D.S., McDaniel, M.R.: Use of contrasts for the evaluation of panel inconsistency. J. Sens. Stud. 5(4), 265–277 (1990)

    Article  Google Scholar 

  • Lundahl, D.S., McDaniel, M.R.: Influence of panel inconsistency on the outcome of sensory evaluations from descriptive panels. J. Sens. Stud. 6(3), 145–157 (1991)

    Article  Google Scholar 

  • Manisera, M., Piccolo, D., Zuccolotto, P.: Analyzing and modelling rating data for sensory analysis in food industry. Quad. Stat. 13, 68–81 (2011)

    Google Scholar 

  • Næs, T., Solheim, R.: Detection and interpretation of variation within and between assessors in sensory profiling. J. Sens. Stud. 6(3), 159–177 (1991)

    Article  Google Scholar 

  • Piggott, J.R.: Design questions in sensory and consumer science. Food Qual. Prefer. 6(4), 217–220 (1995)

    Article  Google Scholar 

  • Pinto, F.S.T., Fogliatto, F.S., Qannari, E.M.: A method for panelists consistency assessment in sensory evaluations based on the cronbachs alpha coefficient. Food Qual. Prefer. 32, 41–47 (2014)

    Article  Google Scholar 

  • Rossi, F.: Assessing sensory panelist performance using repeatability and reproducibility measures. Food Qual. Prefer. 12(5), 467–479 (2001)

    Article  Google Scholar 

  • Schlich, P.: Grapes: a method and a sas® program for graphical representations of assessor performances. J. Sens. Stud. 9(2), 157–169 (1994)

    Article  Google Scholar 

  • Vanacore, A., Pellegrino, M.S.: Characterizing the extent of rater agreement via a non-parametric benchmarking procedure. In: Proceedings of the Conference of the Italian Statistical Society, pp. 999–1004. Italian Statistical Society (2017)

  • Vanbelle, S.: Agreement between raters and groups of raters. Ph.D. thesis, Université de Liège, Belgique (2009)

  • Veall, M.R., Zimmermann, K.F.: Performance measures from prediction–realization tables. Econ. Lett. 39(2), 129–134 (1992)

    Article  Google Scholar 

Download references

Acknowledgements

The authors deeply thank the two anonymous referees for their careful reading of the manuscript and helpful suggestions.

Author information

Authors and Affiliations

  1. Department of Industrial Engineering, University of Naples “Federico II”, p.le Tecchio 80, 80125, Naples, Italy

    Amalia Vanacore & Maria Sole Pellegrino

Authors
  1. Amalia Vanacore
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Maria Sole Pellegrino
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Amalia Vanacore.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Vanacore, A., Pellegrino, M.S. Checking quality of sensory data via an agreement-based approach. Qual Quant 53, 2545–2556 (2019). https://doi.org/10.1007/s11135-018-0807-5

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11135-018-0807-5

Keywords

Search

Navigation