Skip to main content

Ranking vs. Preference: A Comparative Study of Self-reporting

  • Conference paper
Affective Computing and Intelligent Interaction (ACII 2011)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 6974))


This paper introduces a comparative analysis between rating and pairwise self-reporting via questionnaires in user survey experiments. Two dissimilar game user survey experiments are employed in which the two questionnaire schemes are tested and compared for reliable affect annotation. The statistical analysis followed to test our hypotheses shows that even though the two self-reporting schemes are consistent there are significant order of reporting effects when subjects report via a rating questionnaire. The paper concludes with a discussion of the appropriateness of each self-reporting scheme under conditions drawn from the experimental results obtained.

An Erratum for this chapter can be found at .

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Similar content being viewed by others


  1. Agresti, A.: Analysis of ordinal paired comparison data. Journal of the Royal Statistical Society. Series C (Applied Statistics) 41(2), 287–297 (1992)

    MATH  Google Scholar 

  2. Chan, J.C.: Response-order effects in Likert-type scales. Educational and Psychological Measurement 51(3), 531–540 (1991)

    Article  Google Scholar 

  3. Cohen, J.: A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20, 37–46 (1960)

    Article  Google Scholar 

  4. Cronbach, J.L.: Coefficient alpha and the internal structure of tests. Psychometrika 16(3), 297–334 (1951)

    Article  Google Scholar 

  5. Likert, R.: A technique for the measurement of attitudes. Archives of Psychology 140, 1–55 (1932)

    Google Scholar 

  6. Linn, R., Gronlund, N.: Measurement and assessment in teaching. Prentice-Hall, Englewood Cliffs (2000)

    Google Scholar 

  7. Lund, H.H., Klitbo, T., Jessen, C.: Playware technology for physically activating play. Artifical Life and Robotics Journal 9(4), 165–174 (2005)

    Article  Google Scholar 

  8. Poels, K., IJsselsteijn, W.: Development and validation of the game experience questionnaire. In: FUGA Workshop Mini-Symposium, Helsinki, Finland (2008)

    Google Scholar 

  9. Read, J., MacFarlane, S., Cassey, C.: Endurability, engagement and expectations. In: Proceedings of International Conference for Interaction Design and Children (2002)

    Google Scholar 

  10. Scheffe, H.: An analysis of variance for paired comparisons. Journal of the American Statistical Association 47(259), 381–400 (1952)

    MathSciNet  MATH  Google Scholar 

  11. Stevens, S.S.: On the Theory of Scales of Measurement. Science 103(2684), 677–680 (1946)

    Article  MATH  Google Scholar 

  12. Tognetti, S., Garbarino, M., Bonarini, A., Matteucci, M.: Modeling enjoyment preference from physiological responses in a car racing game. In: Proceedings of the IEEE Conference on Computational Intelligence and Games, Copenhagen, Denmark, August 18-21, pp. 321–328 (2010)

    Google Scholar 

  13. Viswanathan, M.: Measurement of individual differences in preference for numerical information. Journal of Applied Psychology 78(5), 741–752

    Google Scholar 

  14. Yannakakis, G.N.: Preference Learning for Affective Modeling. In: Proceedings of the Int. Conf. on Affective Computing and Intelligent Interaction, pp. 126–131. IEEE, Amsterdam (2009)

    Google Scholar 

  15. Yannakakis, G.N., Hallam, J.: Towards Optimizing Entertainment in Computer Games. Applied Artificial Intelligence 21, 933–971 (2007)

    Article  Google Scholar 

  16. Yannakakis, G.N., Hallam, J., Lund, H.H.: Entertainment Capture through Heart Rate Activity in Physical Interactive Playgrounds. User Modeling and User-Adapted Interaction, Special Issue: Affective Modeling and Adaptation 18(1-2), 207–243 (2008)

    Article  Google Scholar 

  17. Yannakakis, G.N., Maragoudakis, M., Hallam, J.: Preference Learning for Cognitive Modeling: A Case Study on Entertainment Preferences. IEEE Systems, Man and Cybernetics; Part A: Systems and Humans 39(6), 1165–1175 (2009)

    Article  Google Scholar 

  18. Yannakakis, G.N., Martínez, H.P., Jhala, A.: Towards Affective Camera Control in Games. User Modeling and User-Adapted Interaction 20(4), 313–340 (2010)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations


Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2011 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Yannakakis, G.N., Hallam, J. (2011). Ranking vs. Preference: A Comparative Study of Self-reporting. In: D’Mello, S., Graesser, A., Schuller, B., Martin, JC. (eds) Affective Computing and Intelligent Interaction. ACII 2011. Lecture Notes in Computer Science, vol 6974. Springer, Berlin, Heidelberg.

Download citation

  • DOI:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-24599-2

  • Online ISBN: 978-3-642-24600-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics