Abstract
Since decades, surveys have been the main source of data in a considerable amount of studies. Designing surveys implies taking many decisions which affect the data quality and thus the results. In this paper, we focus on one of these decisions: the number of answer categories in bipolar closed-ended item specific attitudinal questions. We investigate the measurement quality (product of reliability and validity) of such scales using data from three Multitrait-Multimethod experiments implemented in the European Survey Social (face-to-face): two about social trust (rounds 1 and 4), and one about immigration (round 6). Data are analyzed using the Estimation Using Pooled Data procedure (Saris and Satorra in Struct. Equ. Modeling 25(5): 659–672, 2018). The results show that, out of the three scales tested, the 11-point scale has higher quality in the immigration experiment whereas in the social trust experiments, the 6-point is the one with the highest quality.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
All data can be found at the correspondent webpage of the surveys.
Notes
In other modes like mobile web surveys, we would have different expectations, since 11-point might not fit well on a smartphone screen.
For an explanation of Lisrel notations, see also https://eval-serv2.metpsy.uni-jena.de/wiki-metheval-hp/images/6/61/LISREL_Reference_Sheet_2005_11_10.pdf.
References
Alwin, D.F.: Margins of error: a study of reliability in survey measurement (Vol. 547). John Wiley & Sons (2007)
Alwin, D.F., Krosnick, J.A.: The reliability of survey attitude measurement: the influence of question and respondent attributes. Sociological Methods Res. 20(1), 139–141 (1991)
Andrews, F.M.: Construct validity and error components of survey measures: a structural modeling approach. Public Opin. Q. 48(2), 409–442 (1984)
Bendig, A.W.: Reliability and the number of rating-scale categories. J. Appl. Psychol. 38, 38–40 (1954)
Birkett, N. J. (1986). Selecting the number of response categories for a Likert-type scale. In: Proceedings of the American statistical association. p. 488–492
Bosch, O.J., Revilla, M. (forthcoming). The quality of survey questions in Spain: a cross-national comparison. Revista Española de Investigaciones Sociológicas.
Campbell, D.T., Fiske, D.W.: Convergent and discriminant validation by the multitrait-multimethod matrix. Psychol. Bull. 56(2), 81 (1959)
Cox, E.P., III.: The optimal number of response alternatives for a scale: A review. J. Mark. Res. 17(4), 407–422 (1980)
DeCastellarnau, A.: A classification of response scale characteristics that affect data quality: a literature review. Qual. Quant. 52(4), 1523–1559 (2018)
Dillman, D.A.: 1978 Mail and telephone surveys: the total design method. John Wiley, New York (1978)
Dillman, D.A.: Mail and Internet surveys: the tailored design method–2007 Update with new Internet, visual, and mixed-mode guide. John Wiley & Sons (2011)
Garner, W.R.: Rating scales, discriminability, and information transmission. Psychol. Rev. 67(6), 343 (1960)
Jöreskog, K.G., Sörbom D.: LISREL 8: User’s reference guide. Scientific Software International (1996)
Komorita, S.S.: Attitude content, intensity, and the neutral point on a Likert scale. J. Soc. Psychol. 61(2), 327–334 (1963)
Krosnick, J. A., Fabrigar, L. R. (1997). Designing rating scales for effective measurement in surveys. Survey measurement and process quality, 141–164. https://onlinelibrary.wiley.com/doi/bs/10.1002/9781118490013.ch6?TB_iframe=true&height=658.8&width=370.8
Krosnick, J.A., Presser, S.: Question and questionnaire design. In: Marsden, P.V., Wright, J.D. (eds.) Handbook of survey research, pp. 263–314. Emerald Group Publishing, Bingley, UK (2010)
Lundmark, S., Gilljam, M., Dahlberg, S.: Measuring generalized trust: an examination of question wording and the number of scale points. Public Opin. Q. 80(1), 26–43 (2015)
Masters, E.R.: The relationship between number of response categories and reliability of liker-type questionnaires. J. Educ. Meas. 11(1), 49–53 (1974)
Matell, M.S., Jacoby, J.: Is there an optimal number of alternatives for likert scale items? Study I: reliability and validity. Educ. Psychol. Measur. 31(3), 657–674 (1971)
Preston, C.C., Colman, A.M.: Optimal number of response categories in rating scales: reliability, validity, discriminating power, and respondent preferences. Acta Physiol. (oxf) 104(1), 1–15 (2000)
Revilla, M., Ochoa, C.: Quality of different scales in an online survey in Mexico and Colombia. J. Politics Latin Am. 7(3), 157–177 (2015)
Revilla, M., Saris, W.E.: The split-ballot multitrait-multimethod approach: Implementation and problems. Struct. Equ. Modeling 20(1), 27–46 (2013)
Revilla, M.A., Saris, W.E., Krosnick, J.A.: Choosing the number of categories in agree–disagree scales. Sociological Methods Res. 43(1), 73–97 (2014)
Rodgers, W.L., Andrews, F.M., Regula Herzog, A.: Quality of survey measures: a structural modeling approach. J. Official Statistics-Stockholm- 8, 251–251 (1992)
Saris, W.: The prediction of question quality: the SQP 2.0 software. In: Kleiner, B., Renschler, I., Wernli, B., et al. (eds.) Understanding Research Infrastructures in the Social Sciences, pp. 135–144. Seismo Press, Zurich (2013)
Saris, W.E., Andrews, F.M.: Evaluation of measurement instruments using a structural modeling approach. In: Biemer, P.P., Groves, R.M., Lyberg, L.E., Mathiowetz, N.A., Sudman, S. (eds.) Measurement Errors in Surveys, pp. 575–597. John Wiley & Sons Inc, New York (1991)
Saris, W., Gallhofer, I.N.: Design, evaluation, and analysis of questionnaires for survey research. John Wiley & Sons (2014)
Saris, W.E., Revilla, M.: Correction for measurement errors in survey research: necessary and possible. Soc. Indic. Res. 127(3), 1005–1020 (2016)
Saris, W., Satorra, A.: The pooled data approach for the estimation of split-ballot multitrait–multimethod experiments. Struct. Equ. Modeling 25(5), 659–672 (2018)
Saris, W.E., Satorra, A., Coenders, G.: A new approach to evaluating the quality of measurement instruments: the split-ballot MTMM design. Sociol. Methodol. 34(1), 311–347 (2004)
Saris, W.E., Satorra, A., Van der Veld, W.M.: Testing structural equation models or detection of misspecifications? Struct. Equ. Model. 16(4), 561–582 (2009)
Saris, W., Revilla, M., Krosnick, J.A., Shaeffer, E.M.: Comparing questions with agree/disagree response options to questions with item-specific response options. Surv. Res. Methods 4(1), 61–79 (2010)
Saris, W.E., van Meurs, A.: Evaluation of Measurement Instruments by Meta-Analysis of Multitrait-MultimethodStudies. North Holland, Amsterdam, Netherlands (1990)
Schaeffer, N.C., Presser, S.: The science of asking questions. Ann. Rev. Sociol. 29(1), 65–88 (2003)
Scherpenzeel, A.: Why use 11-point scales. Swiss Household Panel (SHP) 9, 2008 (2002)
Scherpenzeel, A.C., Saris, W.E.: The validity and reliability of survey questions: a meta-analysis of MTMM studies. Sociological Methods Res. 25(3), 341–383 (1997)
Schuman, H., Presser, S.: Questions and answers in attitude surveys: experiments on question form, wording, and context. Sage (1996)
Sturgis, P., Roberts, C., Smith, P.: Middle alternatives revisited: How the neither/nor response acts as a way of saying “I don’t know”? Sociological Methods Res. 43(1), 15–38 (2014)
Sudman, S., Bradburn, N. M., Wansink, B.: Asking questions: The definitive guide to questionnaire design. Jossey-Bass (2004)
Tourangeau, R., Rips, L.J., Rasinski, K.: The psychology of survey response. Cambridge University Press (2000)
van Meurs, A., W. E. Saris. (1990). Memory Effects in MTMM Studies. Evaluation of Measurement Instruments by Meta-analysis of Multitrait-Multimethod Studies, 134–146
Zavala-Rojas, D., Saris, W.E.: Measurement invariance in multilingual survey research: the role of the language of the questionnaire. Soc. Indic. Res. 140(2), 485–510 (2018)
Funding
No special funding was provided for this research.
Author information
Authors and Affiliations
Contributions
All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by Marc Asensio and Melanie Revilla. The first draft of the manuscript was written by Marc Asensio and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Conflict of interest
No conflicts of interest were faced while conducting the research.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendices
Appendix 1 Notations in Lisrel for a True Score model with 3 traits and 3 methods
ga 1 1 = loading between T11 and F1 | |
γ = gamma = ga | ga 2 2 = loading between T21 and F2 |
λ = lambda y = ly | ga 3 3 = loading between T31 and F3 |
θ = teta epsilon = te | ga 4 1 = loading between T12 and F1 |
φ = phi = ph | ga 5 2 = loading between T22 and F2 |
ga 6 3 = loading between T32 and F3 | |
fr = free; fi = fixed | ga 7 1 = loading between T13 and F1 |
va = value; eq = equal | ga 8 2 = loading between T23 and F2 |
ga 9 3 = loading between T33 and F3 | |
te 1 1 = error term associated to Y11 | ga 1 4 = loading between T11 and M1 |
te 2 2 = error term associated to Y22 | ga 2 4 = loading between T21 and M1 |
te 3 3 = error term associated to Y33 | ga 3 4 = loading between T31 and M1 |
te 4 4 = error term associated to Y44 | ga 4 5 = loading between T12 and M2 |
te 5 5 = error term associated to Y55 | ga 5 5 = loading between T22 and M2 |
te 6 6 = error term associated to Y66 | ga 6 5 = loading between T32 and M2 |
te 7 7 = error term associated to Y77 | ga 7 6 = loading between T13 and M3 |
te 8 8 = error term associated to Y88 | ga 8 6 = loading between T23 and M3 |
te 9 9 = error term associated to Y99 | ga 9 6 = loading between T33 and M3 |
ph 1 1 = trait 1 variance | ly 1 1 = loading between T11 and Y11 |
ph 2 2 = trait 2 variance | ly 2 2 = loading between T21 and Y21 |
ph 3 3 = trait 3 variance | ly 3 3 = loading between T31 and Y31 |
ph 4 4 = method 1 variance | ly 4 4 = loading between T12 and Y12 |
ph 5 5 = method 2 variance | ly 5 5 = loading between T22 and Y22 |
ph 6 6 = method 3 variance | ly 6 6 = loading between T32 and Y32 |
ph 2 1 = correlation between traits 1 and 2 | ly 7 7 = loading between T13 and Y13 |
ph 3 1 = correlation between traits 1 and 3 | ly 8 8 = loading between T23 and Y23 |
ph 3 2 = correlation between traits 2 and 3 | ly 9 9 = loading between T33 and Y33 |
Appendix 1a Example of Lisrel input for the Pooled Data MTMM model
! Pooled data group 1.
da ng = 2 ni = 9 no = 12,529 ma = cm.
km file = sb-group-1.corr.
mean file = sb-group-1.mean.
sd file = sb-group-1.sd.
model ny = 9 ne = 9 nk = 6 ly = fu,fi te = di,fi ps = di,fi be = fu,fi ga = fu,fi ph = sy,fi.
va 1 ly 1 1 ly 2 2 ly 3 3 ly 4 4 ly 5 5 ly 6 6.
fr te 1 1 te 2 2 te 3 3 te 4 4 te 5 5 te 6 6.
va 1 te 7 7 te 8 8 te 9 9.
va 0 ly 7 7 ly 8 8 ly 9 9.
fr ga 4 1 ga 7 1 ga 5 2 ga 8 2 ga 6 3 ga 9 3.
va 1 ga 1 1 ga 2 2 ga 3 3.
fr ph 1 1 ph 2 2 ph 3 3 ph 2 1 ph 3 1 ph 3 2 ph 4 4 ph 5 5 ph 6 6.
va 1 ga 1 4 ga 4 5 ga 7 6 ga 2 4 ga 5 5 ga 8 6 ga 3 4 ga 6 5 ga 9 6.
fr ga 3 4 ga 6 5.
out iter = 2000 ns adm = off all sc mi.
Group 2.
da ni = 9 no = 11,977 ma = cm.
km file = sb-group-2.corr.
mean file = sb-group-2.mean.
sd file = sb-group-2.sd.
model ny = 9 ne = 9 nk = 6 ly = fu,fi te = di,fi ps = in be = in ga = in ph = in.
value 1 ly 1 1 ly 2 2 ly 3 3 ly 7 7 ly 8 8 ly 9 9.
fr te 7 7 te 8 8 te 9 9.
eq te 1 1 1 te 1 1.
eq te 1 2 2 te 2 2.
eq te 1 3 3 te 3 3.
va 1 te 4 4 te 5 5 te 6 6.
va 0 ly 4 4 ly 5 5 ly 6 6.
pd.
out iter = 2000 ns adm = off all sc mi.
Appendix 1b Example of Lisrel input for a country split-ballot MTMM model
! ES SPA data group 1.
da ng = 2 ni = 9 no = 426 ma = cm.
km file = sb-group-1.corr.
mean file = sb-group-1.mean.
sd file = sb-group-1.sd.
model ny = 9 ne = 9 nk = 6 ly = fu,fi te = di,fi ps = di,fi be = fu,fi ga = fu,fi ph = sy,fi.
va 1 ly 1 1 ly 2 2 ly 3 3 ly 4 4 ly 5 5 ly 6 6.
fr te 1 1 te 2 2 te 3 3 te 4 4 te 5 5 te 6 6.
va 1 te 7 7 te 8 8 te 9 9.
va 0 ly 7 7 ly 8 8 ly 9 9.
va 1 ga 1 1 ga 2 2 ga 3 3.
fr ph 1 1 ph 2 2 ph 3 3 ph 2 1 ph 3 1 ph 3 2 ph 4 4 ph 5 5 ph 6 6.
!Fixed to values of PDM.
va 0.59 ga 4 1.
va 0.58 ga 5 2.
va 0.57 ga 6 3.
va 0.40 ga 7 1 ga 8 2.
va 0.38 ga 9 3.
va 1.95 ga 2 4.
va 0.96 ga 5 5.
va 0.93 ga 6 5.
va 0.98 ga 9 6.
va 1 ga 1 4 ga 4 5 ga 7 6 ga 8 6 ga 3 4.
out mi iter = 2000 adm = off sc.
Group 2.
da ni = 9 no = 448 ma = cm.
km file = sb-group-2.corr.
mean file = sb-group-2.mean.
sd file = sb-group-2.sd.
model ny = 9 ne = 9 nk = 6 ly = fu,fi te = di,fi ps = in be = in ga = in ph = in.
va 1 ly 1 1 ly 2 2 ly 3 3 ly 7 7 ly 8 8 ly 9 9.
eq te 1 1 1 te 1 1.
eq te 1 2 2 te 2 2.
eq te 1 3 3 te 3 3.
fr te 7 7 te 8 8 te 9 9.
va 1 te 4 4 te 5 5 te 6 6.
va 0 ly 4 4 ly 5 5 ly 6 6.
pd.
out mi iter = 2000 adm = off sc.
Appendix 2 Parameters set free in the Pooled Data Model analysis (using Lisrel notations)
Experiment | Result of BM | Final model |
|---|---|---|
Social Trust R1 | IS | BM + fr ga 3 4 |
Social Trust R4 | PS | BM + fr ga 6 6 + eq ga 1 4 ga 3 5 ga 6 6 |
Evaluation of Immigration | IS | BM + fr ga 2 4 + ga 5 5 + ga 6 5 + ga 9 6 |
Appendix 3 Parameters fixed in the country (-language) analysis for each experiment (in Lisrel notations) and their values
Experiment | Parameters fixed and their values |
|---|---|
Social Trust R1 | va 0.60 ga 4 1 |
va 0.58 ga 5 2 | |
va 0.57 ga 6 3 | |
va 0.18 ga 7 1 ga 9 3 | |
va 0.19 ga 8 2 | |
va 0.64 ga 3 4 | |
Social Trust R4 | va 0.18 ga 3 1 |
va 0.19 ga 4 2 | |
va 0.33 ga 5 3 | |
va 0.62 ga 6 1 | |
va 0.55 ga 7 2 | |
va 0.98 ga 8 3 | |
va 0.76 ga 1 4 ga 3 5 ga 6 6 | |
Evaluation of Immigration | va 0.59 ga 4 1 |
va 0.58 ga 5 2 | |
va 0.57 ga 6 3 | |
va 0.40 ga 7 1 ga 8 2 | |
va 0.38 ga 9 3 | |
va 1.95 ga 2 4 | |
va 0.96 ga 5 5 | |
va 0.93 ga 6 5 | |
va 0.98 ga 9 6 |
Appendix 4a Summary of the Social Trust 1 experiment
Country | Sample Size | Result of BM | Final model |
|---|---|---|---|
Pooled Data model | IS | BM + fr ga 3 4 | |
Austria | 2,250 | PS | BM |
Belgium | 643 | PS | BM |
Switzerland | 657 | PS | BM + fr ga 5 2 ga 7 1 ga 9 3 |
Czech Republic | 1,302 | PS | BM + fr ga 6 3 ga 8 2 ga 9 3 |
Germany | 935 | PS | BM + fr ga 9 3 |
Denmark | 1,466 | NC | BM + fr ga 3 4 ga 8 2 ga 9 3 |
Spain | 582 | IS | BM + fr ga 9 3 |
Finland | 1,774 | PS | BM + fr ga 7 1 ga 9 3 |
France | 1,350 | PS | BM |
Great Britain | 1,777 | PS | BM + fr ga 3 4 ga 9 3 |
Greece | 2,564 | IS | BM + fr ga 7 1 ga 9 3 |
Ireland | 664 | PS | BM |
Israel | 817 | PS | BM + fr ga 7 1 ga 9 3 |
Netherlands | 2,334 | PS | BM + fr ga 7 1 ga 8 2 ga 9 3 |
Norway | 605 | PS | BM + fr ga 3 3 ga 8 2 |
Poland | 2,098 | PS | BM + fr ga 3 3 ga 7 1 ga 9 3 |
Portugal | 499 | PS | BM + fr ga 8 2 |
Sweden | 1,692 | PS | BM + fr ga 9 3 |
Slovenia | 495 | PS | BM |
Appendix 4b Summary of the Social Trust 4 experiment
Country | Language | Sample size | Result of BM | Final model |
|---|---|---|---|---|
Pooled Data model | - | IS | BM + fr ga 6 6 + eq ga 6 6 ga 1 5 ga 3 5 | |
Belgium | Dutch | 712 | PS | BM + fr ga 3 1 |
Belgium | French | 468 | PS | BM + fr ga 6 6 |
Bulgaria | Bulgarian | 1,455 | IS | BM + fr ga 1 1 |
Switzerland | French | 258 | PS | BM + fr ga 3 1 |
Switzerland | German | 940 | PS | BM + fr ga 3 1 ga 4 2 |
Cyprus | Greek | 769 | PS | BM + fr ga 4 2 ga 7 2 |
Czech Republic | Czech | 1,339 | IS | BM + fr ga 1 1 ga 8 3 |
Germany | German | 1,858 | IS | BM + fr ga 1 1 ga 7 2 ga 5 3 |
Denmark | Danish | 1,054 | PS | BM + fr ga 4 2 ga 7 2 ga 8 3 |
Estonia | Estonian | 721 | PS | BM + fr ga 2 2 |
Estonia | Russian | 294 | PS | BM |
Spain | Spanish | 1,142 | PS | BM + fr ga 5 3 ga 7 2 |
Finland | Finnish | 412 | PS | BM + fr ga 1 1 ga 3 1 |
France | French | 1,487 | PS | BM + ga 5 3 ga 6 6 ga 8 6 |
Great Britain | English | 1,581 | PS | BM + fr ga 5 3 ga 8 6 |
Greece | Greek | 1,376 | PS | BM + fr ga 5 3 ga 8 6 |
Croatia | Croatian | 1,024 | PS | BM + fr ga 7 2 ga 3 5 |
Israel | Arab | 196 | PS | BM |
Israel | Hebrew | 1,295 | PS | BM + fr ga 3 5 ga 6 6 ga 7 2 |
Latvia | Latvian | 929 | PS | BM + fr ga 3 1 ga 3 5 |
Latvia | Russian | 363 | PS | BM |
Netherlands | Dutch | 1,004 | PS | BM + fr ga 8 6 |
Norway | Norwegian | 553 | IS | BM + fr ga 3 1 ga 4 2 ga 6 6 ga 8 3 |
Poland | Polish | 1,079 | PS | BM + fr ga 6 6 |
Portugal | Portuguese | 1,609 | PS | BM + fr ga 3 5 |
Romania | Romanian | 1,402 | PS | BM + fr ga 3 1 ga 4 2 ga 8 3 |
Russia | Russian | 1,645 | PS | BM + fr ga 2 2 ga 5 3 |
Sweden | Swedish | 397 | PS | BM + fr ga 4 2 ga 3 5 |
Slovenia | Slovene | 845 | PS | BM |
Slovakia | Slovak | 1,124 | PS | BM + fr ga 3 5 ga 5 3 |
Turkey | Turkish | 1,589 | IS | BM + fr ga 3 1 ga 5 3 ga 7 2 |
Ukraine | Russian | 630 | PS | BM |
Ukraine | Ukrainian | 572 | PS | BM + fr ga 3 5 ga 6 6 |
Appendix 4c Summary of the Evaluation of the Immigration experiment
Country | Language | Sample size | Result of BM | Final model |
|---|---|---|---|---|
Pooled Data model | - | IS | BM + fr ga 2 4 ga 5 5 ga 6 5 ga 9 6 | |
Belgium | Dutch | 369 | IS | BM + fr ga 1 1 ga 3 3 |
Belgium | French | 521 | IS | BM + fr ga 3 3 ga 8 2 |
Bulgaria | Bulgarian | 998 | PS | BM + fr ga 1 4 |
Switzerland | French | 161 | IS | BM + fr ga 1 4 ga 2 2 + eq ga 1 4 ga 4 5 ga 7 6 |
Switzerland | German | 510 | IS | BM + fr ga 1 1 ga 1 4 |
Cyprus | Greek | 552 | PS | BM + fr ga 1 4 |
Czech Republic | Czech | 940 | PS | BM |
Germany | German | 1,483 | PS | BM |
Denmark | Danish | 820 | PS | BM |
Estonia | Estonian | 831 | PS | BM |
Estonia | Russian | 344 | PS | BM + fr ga 9 6 |
Spain | Spanish | 874 | PS | BM + fr ga 9 6 |
Finland | Finnish | 1,042 | PS | BM |
France | French | 998 | PS | BM + fr ga 5 5 |
Ireland | English | 1,306 | PS | BM |
Hungary | Hungarian | 956 | PS | BM + fr ga 1 4 ga 2 4 |
Great Britain | English | 1,042 | PS | BM |
Israel | Arab | 163 | PS | BM + fr ga 1 4 |
Israel | Hebrew | 926 | PS | BM |
Iceland | Icelandic | 361 | IS | BM + fr ga 3 3 ga 6 5 |
Italy | Italian | 456 | IS | BM + fr ga 3 3 ga 5 5 |
Lithuania | Lithuanian | 961 | PS | BM |
Netherlands | Dutch | 906 | IS | BM + fr ga 1 4 ga 4 5 ga 5 2 |
Norway | Norwegian | 778 | PS | BM |
Poland | Polish | 916 | PS | BM + fr ga 1 4 |
Portugal | Portuguese | 843 | PS | BM + fr ga 1 4 ga 6 5 |
Russia | Russian | 1,195 | PS | BM |
Sweden | Swedish | 891 | PS | BM + fr ga 1 4 ga 5 5 |
Slovenia | Slovene | 628 | PS | BM + fr ga 1 4 |
Slovakia | Slovak | 833 | PS | BM + fr ga 6 5 |
Ukraine | Russian | 454 | PS | BM + fr ga 1 4 ga 6 5 |
Ukraine | Ukrainian | 555 | PS | BM + fr ga 1 4 |
Rights and permissions
About this article
Cite this article
Asensio, M., Revilla, M. Number of answer categories for bipolar item specific scales in face-to-face surveys: Does more mean better?. Qual Quant 56, 1413–1433 (2022). https://doi.org/10.1007/s11135-021-01183-x
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11135-021-01183-x