Advertisement

Die Ergänzung kognitiver Interviews um Eye Tracking

Ein Methodenvergleich
  • Cornelia E. NeuertEmail author
  • Timo Lenzner
Chapter
Part of the Schriftenreihe der ASI - Arbeitsgemeinschaft Sozialwissenschaftlicher Institute book series (SASI)

Zusammenfassung

Umfrageforschern stehen eine Vielzahl von Methoden zur Verfügung, um Survey-Fragen zu evaluieren. Dabei stellt sich die Frage, welche Methoden am effektivsten sind, um mögliche Probleme in Erhebungsinstrumenten festzustellen. Als Beitrag zu dieser zentralen Frage wurde in einem Methodenvergleich untersucht, ob eine Ergänzung kognitiver Interviews um Eye Tracking wirksamer ist beim Identifizieren von Problemen in Fragen als die Methode des kognitiven Interviews allein. Verglichen werden die Gesamtanzahl der gefundenen Probleme und die Anzahl der Fragen, die als fehlerhaft identifiziert wurden. Die Ergebnisse zeigen, dass sich die beiden Methoden Eye Tracking und kognitives Interview effektiv ergänzen. Durch den Einsatz der hybriden Methode wurden im Vergleich zum alleinigen Einsatz des kognitiven Interviews mehr Probleme aufgedeckt und mehr Fragen als problematisch identifiziert.

Schlüsselwörter

Kognitives Interview Eye Tracking Pretesting Erhebungsinstrumente 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Literatur

  1. Beatty, P. C., & Willis, G. B. (2007). Research synthesis: The practice of cognitive interviewing. Public Opinion Quarterly, 71(2), 287-311.  https://doi.org/10.1093/poq/nfm006CrossRefGoogle Scholar
  2. Blair, J., & Conrad, F.G. (2011). Sample size for cognitive interview pretesting. Public Opinion Quarterly, 75(4), 636-658.  https://doi.org/10.1093/poq/nfr035CrossRefGoogle Scholar
  3. Blair, J., & Srinath, K. P. (2008). A note on sample size for behavior coding pretests. Field Methods, 20(1), 85-95.  https://doi.org/10.1177/1525822x07303601CrossRefGoogle Scholar
  4. Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment 6(4), 284–290.CrossRefGoogle Scholar
  5. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37-46.CrossRefGoogle Scholar
  6. Collins, D. (2003). Pretesting survey instruments: an overview of cognitive methods. Quality of Life Research, 12(3), 229-238.CrossRefGoogle Scholar
  7. Conrad, F. G., & Blair, J. (2001). Interpreting verbal reports in cognitive interviews: Probes matter. Proceedings of the Section on Survey Research Methods, American Statistical Association.Google Scholar
  8. Conrad, F. G., & Blair, J. (2004). Data quality in cognitive interviews: the case of verbal reports. In S. Presser, J. M. Rothgeb, M. P. Couper, J. T. Lessler, E. Martin, J. Martin, & E. Singer (Eds.), Methods for testing and evaluating survey questionnaires (67-87). New York: Wiley.  https://doi.org/10.1002/0471654728.ch4
  9. Conrad, F. G., & Blair, J. (2009). Sources of error in cognitive interviews. Public Opinion Quarterly, 73(1), 32-55.CrossRefGoogle Scholar
  10. Conrad, F. G., Blair, J. & Tracy, E. (1999). Verbal reports are data! A theoretical approach to cognitive interviews. In Proceedings of the Federal Committee on Statistical Methodology Research Conference. Arlington, VA, pp. 11-20.Google Scholar
  11. DeMaio, T. J., & Landreth, A. (2004). Do different cognitive interview techniques produce different results? In S. Presser, J. M. Rothgeb, M. P. Couper, J. T. Lessler, E. Martin, J. Martin, & E. Singer (Eds.), Methods for testing and evaluating survey questionnaires (89-108). New York: Wiley.  https://doi.org/10.1002/0471654728.ch5
  12. Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behaviour Research Methods, 39, 175-191.CrossRefGoogle Scholar
  13. Forsyth, B. H., & Lessler, J. T. (1991). Cognitive laboratory methods: a taxonomy. In P. P. Biemer, R. M. Groves, L. E. Lyberg, N. A. Mathiowetz, & S. Sudman (Eds.), Measurement Errors in Surveys (393-418). New York: Wiley.Google Scholar
  14. Fowler, F. J. (1992). How unclear terms affect survey data. Public Opinion Quarterly 56(2), 218-31.  https://doi.org/10.1086/269312
  15. Fowler, F. J. (1995). Improving survey questions. Design and evaluations. Thousand Oaks: Sage.Google Scholar
  16. Fowler, F. J., & Cannell, C. F. (1996). Using behavioral coding to identify cognitive problems with survey questions. In N. Schwarz & S. Schuman (Eds.), Answering questions. Methodology for determining cognitive and communicative processes in survey research (15-36). San Francisco: Jossey-Bass.Google Scholar
  17. Galesic, M., Tourangeau, R., Couper, M. P., & Conrad, F.G. (2008). Eye-Tracking data: New insights on response order effects and other cognitive shortcuts in survey responding. Public Opinion Quarterly, 72(5), 892-913.  https://doi.org/10.1093/poq/nfn059CrossRefGoogle Scholar
  18. Galesic, M., & Yan, T. (2011). Use of eye tracking for studying survey response processes. In M. Das, P. Ester, & L. Kaczmirek (Eds.), Social and behavioral research and the internet: Advances in applied methods and research strategies (349-370). New York: Routledge.Google Scholar
  19. Graesser, A. C., Cai, Z., Louwerse, M. M., & Daniel, F. (2006). Question understanding aid (QUAID). A web facility that tests question comprehensibility. Public Opinion Quarterly, 70(1), 3–22.  https://doi.org/10.1093/poq/nfj012CrossRefGoogle Scholar
  20. Groves, R. M., Fowler Jr, F. J., Couper, M. P., Lepkowski, J. M., Singer, E., & Tourangeau, R. (2004). Survey Methodology. Hoboken, NJ: John Wiley & Sons.Google Scholar
  21. Holmqvist, K., Holsanova, J., Barthelson, M., & Lundqvist, D. (2003). Reading or scanning? A study of newspaper and net paper reading. In J. Hyöna, R. Radach, & H. Deubel (Eds.), The mind’s eye. Cognitive and applied aspects of eye movement research (657-670). Amsterdam: North-Holland.CrossRefGoogle Scholar
  22. Just, M. A., & Carpenter, P.A. (1980). A theory of reading: From eye fixations to comprehension. Psychological Review, 87, 329–354.CrossRefGoogle Scholar
  23. Landis, J. R., & Koch, G.G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174.CrossRefGoogle Scholar
  24. Lenzner, T., Kaczmirek, L., & Galesic, M. (2011). Seeing through the eyes of the respondent: An eye-tracking study on survey question comprehension. International Journal of Public Opinion Research, 23(3), 361-73.  https://doi.org/10.1093/ijpor/edq053CrossRefGoogle Scholar
  25. Lenzner, T., Kaczmirek, L., & Galesic, M. (2014). Left feels right: A usability study on the position of answer boxes in web surveys. Social Science Computer Review, 32(6), 743-764.  https://doi.org/10.1177/0894439313517532CrossRefGoogle Scholar
  26. Menold, N., Kaczmirek, L., Lenzner, T., & Neusar, A. (2014). How do respondents attend to verbal labels in rating scales? Field Methods, 26(1), 21-39.  https://doi.org/10.1177/1525822x13508270CrossRefGoogle Scholar
  27. Neuert, C. E., & Lenzner, T. (2016). Incorporating eye tracking into cognitive interviewing to pretest survey questions. International Journal of Social Research Methodology, 19(5), 501-519.CrossRefGoogle Scholar
  28. Miller, K. (2011). Cognitive interviewing. In J. Madans, K. Miller, A. Maitland, G. Willis (Eds.), Question evaluation methods: Contributing to the science of data quality (51-75). New York: Wiley.Google Scholar
  29. Presser, S., & Blair, J. (1994). Survey pretesting: Do different methods produce different results. Sociological methodology, 24(1), 73-104.CrossRefGoogle Scholar
  30. Presser, S., Couper, M. P, Lessler, J. T., Martin, E., Martin, J., Rothgeb, J. M., & Singer, E. (2004). Methods for testing and evaluating survey questions. Public Opinion Quarterly, 68(1), 109–130.  https://doi.org/10.1093/poq/nfh008CrossRefGoogle Scholar
  31. Rayner, K. (1998). Eye movements in reading and information processing: 20 years of research. Psychological Bulletin, 124, 372–422.CrossRefGoogle Scholar
  32. Redline, C. D., & Lankford, C. P. (2001, May). Eye-movement analysis: a new tool for evaluating the design of visually administered instruments (paper and web). Paper prepared for presentation at the annual meeting of the American Association for Public Opinion Research, Montreal.Google Scholar
  33. Romano, J. C., & Chen, J. M. (2011). A usability and eye-tracking evaluation of four versions of the online national survey of college graduates (NSCG): Iteration 2. Study Series: Survey Methodology 2011-01, Washington D.C.: U.S. Census Bureau.Google Scholar
  34. Tourangeau, R. (1984). Cognitive science and survey methods. In T.B. Jabine, M. L. Straf, J. M. Tanur, & R. Tourangeau (Eds.), Cognitive aspects of survey methodology: Building a bridge between disciplines (73-100). Washington, DC: National Academy Press.Google Scholar
  35. Tourangeau, R., Rips, L. J., & Rasinski, K. (2000). The psychology of survey response. New York: Cambridge University Press.Google Scholar
  36. Tries, S. (2010). Usability tests of online questionnaires. In Federal Statistical Office (Ed.), Methods, Approaches, Developments: Information of the German Federal Statistical Office (5-8). Wiesbaden: Federal Statistical Office.Google Scholar
  37. Van den Haak, M., De Jong, M., & Schellens, P. J. (2003). Retrospective vs. concurrent think-aloud protocols: testing the usability of an online library catalogue. Behaviour & Information Technology, 22(5), 339-351.  https://doi.org/10.1080/0044929031000CrossRefGoogle Scholar
  38. Willis, G. B. (2004). Cognitive interviewing revisited: A useful technique, in theory? In S. Presser, J. M. Rothgeb, M. P. Couper, J. T. Lessler, E. Martin, J. Martin, & E. Singer (Eds.), Methods for testing and evaluating survey questionnaires (23-44). New York: Wiley.  https://doi.org/10.1002/0471654728.ch2
  39. Willis, G. B. (2005). Cognitive interviewing: A tool for improving questionnaire design. London: Sage.Google Scholar

Copyright information

© Springer Fachmedien Wiesbaden GmbH, ein Teil von Springer Nature 2019

Authors and Affiliations

  1. 1.MannheimDeutschland

Personalised recommendations