Skip to main content
SpringerLink
Log in

Can response variance effectively identify careless respondents to multi-item, unidimensional scales?

  • Original Article
  • Published:
Journal of Marketing Analytics Aims and scope Submit manuscript

Abstract

Because careless responding to questionnaire items can be quite common, and even low rates of careless responding can substantially impact some analyzes, researchers have been advised to remove careless respondents before data analysis. However, identifying these respondents is a non-trivial task. In the context of multi-item, unidimensional scales, it has been suggested that response variance may hold information about careless responding. This notion was tested in the current research with two studies. Data collected from respondents indicates that people responding carelessly display more variance in their responses than people responding with more effort. Given this finding, a procedure which uses measures of response variance and cluster analysis to identify careless respondents was developed. The effectiveness of the procedure with different specifications was tested with simulated data and validated with data from actual respondents. On the basis of the results, we advocate using the procedure with specifications that conservatively identify careless respondents. Such an approach will identify the most extreme careless respondents, while maximizing the retention of careful, honest respondents. We discuss the advantages the developed procedure has over existing procedures for identifying careless respondents.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Baer, R.A., Ballenger, J., Berry, D.T.R. and Wetter, M.W. (1997) Detection of random responding on the MMPI-A. Journal of Personality Assessment 68 (1): 139–151.

    Article  Google Scholar 

  • Barbara, D., Domeniconi, C. and Rogers, J.P. (2006) Detecting outliers using transduction and statistical testing. Proceedings of the Association for Computing Machinery 12: 55–64.

    Google Scholar 

  • Baumgartner, H. and Steenkamp, J.E.M. (2001) Response styles in marketing research: A cross-national investigation. Journal of Marketing Research 38 (2): 143–156.

    Article  Google Scholar 

  • Baumgartner, H. and Steenkamp, J.E.M. (2006) An extended paradigm for measurement analysis of marketing constructs applicable to panel data. Journal of Marketing Research 43 (3): 431–442.

    Article  Google Scholar 

  • Beach, D.A. (1989) Identifying the random responder. Journal of Psychology 123 (1): 101–103.

    Article  Google Scholar 

  • Ben-Gal, I. (2005) Outlier detection. In: O. Maimon and L. Rockach (eds.) Data Mining and Knowledge Discovery Handbook: A Complete Guide for Practitioners and Researchers. Norwell, MA: Kluwer Academic Publishers, pp. 1–16.

    Google Scholar 

  • Berry, D.T.R., Wetter, M.W., Baer, R.A., Larsen, L.H., Clark, C. and Monroe, K. (1992) MMPI-2 random-responding indices: Validation using a self-report methodology. Psychological Assessment 4 (3): 340–345.

    Article  Google Scholar 

  • Berry, D.T.R. et al. (1991) Detection of random responding on the MMPI-2: Utility of F, back F, and VRIN scales. Psychological Assessment: A Journal of Consulting and Clinical Psychology 3 (3): 418–423.

    Article  Google Scholar 

  • Crede, M. (2010) Random-responding as a threat to the validity of effect size estimates in correlational research. Educational and Psychological Measurement 70 (4): 596–612.

    Article  Google Scholar 

  • Diamantopoulos, A., Reynolds, N.L. and Simintiras, A.C. (2006) The impact of response styles on the stability of cross-national comparisons. Journal of Business Research 59 (8): 925–935.

    Article  Google Scholar 

  • Gallen, R.T. and Berry, D.T.R. (1996) Detection of random-responding in MMPI-2 protocols. Assessment 3 (2): 171–178.

    Google Scholar 

  • Gallen, R.T. and Berry, D.T.R. (1997) Partially random MMPI-2 protocols: When are they interpretable? Assessment 4 (1): 61–68.

    Google Scholar 

  • Gordon, A.D. (1999) Classification. 2nd edn. Boca Raton, FL: Chapman and Hall.

    Google Scholar 

  • Greenleaf, E.A. (1992) Improving rating scale measures by detecting and correcting bias components in some response styles. Journal of Marketing Research 29 (2): 176–188.

    Article  Google Scholar 

  • Jackson, D.N. (1967) Acquiescence response styles: Problems of identification and control. In I. A. Berg (ed) Response Set in Personality Assessment, Chicago, IL: Aldine Publishing Company, pp. 71–114.

    Google Scholar 

  • Johnson, J.A. (2005) Ascertaining the validity of individual protocols from web-based personality inventories. Journal of Research in Personality 39 (1): 103–129.

    Article  Google Scholar 

  • Karabatsos, G. (2003) Comparing the aberrant response detection performance of thirty-six person-fit statistics. Applied Measurement in Education 16 (4): 277–298.

    Article  Google Scholar 

  • Kilsheimer, J.C., Goldsmith, R.E. and Flynn, L.R. (1992) Status consumption: The concept and its measure. AMA Summer Marketing Educators’ Proceedings 3: 341–342.

    Google Scholar 

  • Lichtenstein, D.R., Ridgway, N.M. and Netemeyer, R.G. (1993) Price perceptions and consumer shopping behavior: A field study. Journal of Marketing Research 30 (2): 234–245.

    Article  Google Scholar 

  • Lichtenstein, D.R., Netemeyer, R.G. and Burton, S. (1995) Assessing the domain specificity of deal proneness: A field study. Journal of Consumer Research 22 (3): 314–326.

    Article  Google Scholar 

  • Meade, A.W. and Craig, S.B. (2012) Identifying careless responses in survey data. Psychological Methods 17 (3): 437–455.

    Article  Google Scholar 

  • Obermiller, C. and Spangenberg, E.R. (1998) Development of a scale to measure consumer skepticism toward advertising. Journal of Consumer Psychology 1 (2): 159–186.

    Article  Google Scholar 

  • Paulhus, D.L. (1991) Measurement and control of response bias. In: J. P. Robinson, P. R. Shaver, and L. S. Wrightsman (eds) Measures of Personality and Social Psychological Attitudes, San Diego, CA: Academic Press, pp. 17–59.

    Chapter  Google Scholar 

  • Peterson, R.A. (1994) A meta-analysis of Cronbach’s coefficient alpha. Journal of Consumer Research 21 (2): 381–391.

    Article  Google Scholar 

  • Richard, M.O. and Toffoli, R. (2009) Language influence in responses to questionnaires by bilingual respondents: A test of the Whorfian hypothesis. Journal of Business Research 62 (10): 987–994.

    Article  Google Scholar 

  • Schaninger, C.M. and Buss, W.C. (1986) Removing response-style effects in attribute-determinance ratings to identify market segments. Journal of Business Research 14 (3): 237–252.

    Article  Google Scholar 

  • Shimp, T.A. and Sharma, S. (1987) Consumer ethnocentrism: Construction and validation of the Cetscale. Journal of Marketing Research 24 (3): 280–289.

    Article  Google Scholar 

  • Swain, S.D., Weathers, D. and Niedrich, R.W. (2008) Assessing three sources of misresponse to reversed likert items. Journal of Marketing Research 45 (1): 116–131.

    Article  Google Scholar 

  • Taleb, N.N. (2004) Fooled by Randomness: The Hidden Role of Change in Life and in the Markets, New York: Random House.

    Google Scholar 

  • Van Rosmalen, J., van Herk, H. and Groenen, P.J.F. (2010) Identifying response styles: A latent-class bilinear multinomial logit model. Journal of Marketing Research 47 (1): 157–172.

    Article  Google Scholar 

  • Ward Jr, J.H. (1963) Hierarchical grouping to optimize an objective function. Journal of the American Statistical Association 58 (301): 236–244.

    Article  Google Scholar 

  • Wilson, E.J. (1995) Research design effects on the reliability of rating scales in marketing: An update on Churchill and Peter. In: F. R. Kardes and M. Sujan (eds) Advances in Consumer Research 22. Provo, UT: Association for Consumer Research, pp. 360–365.

    Google Scholar 

  • Woods, C.M. (2006) Careless responding to reverse-worded items: Implications for confirmatory factor analysis. Journal of Psychopathology and Behavioral Assessment 28 (3): 186–191.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Marketing, College of Business and Behavioral Sciences, Clemson University, Clemson, SC 29634, USA

    Danny Weathers

Authors
  1. Danny Weathers
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ahmet Bardakci
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Danny Weathers.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Weathers, D., Bardakci, A. Can response variance effectively identify careless respondents to multi-item, unidimensional scales?. J Market Anal 3, 96–107 (2015). https://doi.org/10.1057/jma.2015.6

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1057/jma.2015.6

Keywords

Search

Navigation