Skip to main content
SpringerLink
Log in

Evaluating Item-Sort Task Methods: The Presentation of a New Statistical Significance Formula and Methodological Best Practices

  • Published:
Journal of Business and Psychology Aims and scope Submit manuscript

Abstract

Purpose

An item-sort task is a common method to reduce over-representative item lists during the scale-creation process. The current article delineates the limitations and misapplications of the accepted statistical significance formula for item-sort tasks and proposes a new statistical significance formula with greater utility across a wider range of item-sort tasks.

Design

First, a simulation study compares the two formulas in an array of conditions that vary on sample size and number of assignment choices. Second, an empirical study compares the results of three separate item-sort tasks across the two formulas for statistical significance.

Findings

In the empirical study, the proposed formula produces more correct retention decisions than the existing formula across all three item-sort tasks. In the simulation study, the proposed formula is more appropriate than the existing formula under most conditions. The two formulas function identically in item-sort tasks with only two assignment choices.

Implications

Researchers could obtain erroneous results when misapplying the existing item-sort task statistical significance formula to cases with more than two assignment choices. The proposed formula corrects this limitation, ultimately providing accurate results more often than the existing formula. Applying the proposed formula could help future research and practice throughout the scale development process.

Originality

Despite widespread use, few attempts have been made to improve scale-creation pretest methods, particularly item-sort tasks. The current study demonstrates that even conventional statistical methods are susceptible to misuse and misapplication, and future research could benefit from the reexamination of other common methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  • Aday, L. A., & Cornelius, L. J. (2011). Designing and conducting health surveys: A comprehensive guide. Hoboken, NJ: Wiley.

    Google Scholar 

  • Aguinis, H., Werner, S., Abbott, J., Angert, C., Park, J. H., & Kohlhausen, D. (2010). Customer-centric science: Reporting significant research results with rigor, relevance, and practical impact in mind. Organizational Research Methods, 13, 515–539. doi:10.1177/1094428109333339.

    Article  Google Scholar 

  • Anderson, J. C., & Gerbing, D. W. (1991). Predicting the performance of measures in a confirmatory factor analysis with a pretest assessment of their substantive validities. Journal of Applied Psychology, 76, 732–740. doi:10.1037/0021-9010.76.5.732.

    Article  Google Scholar 

  • Anderson, J. C., Håkansson, H., & Johanson, J. (1994). Dyadic business relationships within a business network context. Journal of Marketing, 58, 1–15.

    Article  Google Scholar 

  • Bansal, P. (2005). Evolving sustainably: A longitudinal study of corporate sustainable development. Strategic Management Journal, 26, 197–218.

    Article  Google Scholar 

  • Bauer, T. N., Truxillo, D. M., Sanchez, R. J., Craig, J. M., Ferrara, P., & Campion, M. A. (2001). Applicant reactions to selection: Development of the selection procedural justice scale (SPJS). Personnel Psychology, 54, 387–419.

    Article  Google Scholar 

  • Collins, D. (2003). Pretesting survey instruments: An overview of cognitive methods. Quality of Life Research, 12, 229–238.

    Article  PubMed  Google Scholar 

  • Comrey, A. L., & Lee, H. B. (1992). A first course in factor analysis. Hillsdale, NJ: Lawrence Erlbaum Associates.

    Google Scholar 

  • Conrad, F. G., & Blair, J. (2004). Aspects of data quality in cognitive interviews: The case of verbal reports. In S. Presser, J. M. Rothgeb, M. P. Couper, J. T. Lessler, E. Martin, J. Martin, & E. Singer (Eds.), Methods for testing and evaluating survey questionnaires (pp. 67–88). New York: Wiley.

    Chapter  Google Scholar 

  • Costello, A. B., & Osbourne, J. (2005). Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Practical Assessment, Research & Evaluation, 10, 1–9.

    Google Scholar 

  • Czaja, R. (1998). Questionnaire pretesting comes of age. Marketing Bulletin-Department of Marketing Massey University, 9, 52–66.

    Google Scholar 

  • Dahling, J. J., Whitaker, B. G., & Levy, P. E. (2009). The development and validation of a new Machiavellianism scale. Journal of Management, 35, 219–257.

    Article  Google Scholar 

  • DeMaio, T. J., & Landreth, A. (2004). Do different cognitive interview techniques produce different results? In S. Presser, J. M. Rothgeb, M. P. Couper, J. T. Lessler, E. Martin, J. Martin, & E. Singer (Eds.), Methods for testing and evaluating survey questionnaires. NJ: Wiley, Hoboken. doi:10.1002/0471654728.ch5.

    Google Scholar 

  • Devlin, A., & Jacobs, M. (2010). Antitrust error. William and Mary Law Review, 52, 75–132.

    Google Scholar 

  • Fassnaght, M., & Ibrahim, K. (2006). Quality of electronic services: Conceptualizing and testing a hierarchical model. Journal of Service Research, 9, 19–37.

    Article  Google Scholar 

  • Ferris, D. L., Brown, D. J., Berry, J. W., & Lian, H. (2008). The development and validation of the Workplace Ostracism Scale. Journal of Applied Psychology, 93, 1348–1366. doi:10.1037/a0012743.

    Article  PubMed  Google Scholar 

  • Fiore, A. M., Niehm, L. S., Hurst, J. L., Son, J., & Sadachar, A. (2013). Entrepreneurial marketing: Scale validation with small, independently-owned businesses. Journal of Marketing Development and Competitiveness, 7, 63–86.

    Google Scholar 

  • Geyskens, I., & Steenkamp, J. M. (2000). Economic and social satisfaction: Measurement and relevance to marketing channel celationships. Journal of Retailing, 76, 11–32.

    Article  Google Scholar 

  • Google Scholar (2014). Articles citing “Predicting the performance of measures in a confirmatory factor analysis with a pretest assessment of their substantive validities.” http://scholar.google.com/scholar?cites=6941685027650661312&as_sdt=5,39&sciodt=0,39&hl=en.

  • Gopalakrishnan, S., & Bierly, P. (2001). Analyzing innovation adoption using a knowledge-based approach. Journal of Engineering and Technology management, 18(2), 107–130.

    Article  Google Scholar 

  • Groves, R. M., Fowler, F. J, Jr, Couper, M. P., Lepkowski, J. M., Singer, E., & Tourangeau, R. (2013). Survey methodology. Hoboken, NJ: Wiley.

    Google Scholar 

  • Hardesty, D. M., & Bearden, W. O. (2004). The use of expert judges in scale development: Implications for improving face validity of measures of unobservable constructs. Journal of Business Research, 57, 98–107.

    Article  Google Scholar 

  • Hardy, B., & Ford, L. (2014). It’s not me, it’s you: Miscomprehension in surveys. Organizational Research Methods, 17, 138–162.

    Article  Google Scholar 

  • Hemphill, J. K., & Westie, C. M. (1950). The measurement of group dimensions. The Journal of Psychology, 29, 325–342.

    Article  Google Scholar 

  • Hibbard, J. D., Kumar, N., & Stern, L. W. (2001). Examining the impact of destructive acts in marketing channel relationships. Journal of Marketing Research, 38, 45–61.

    Article  Google Scholar 

  • Hinkin, T. R. (1995). A review of scale development practices in the study of organizations. Journal of Management, 21, 967–988.

    Article  Google Scholar 

  • Hinkin, T. R. (1998). A brief tutorial on the development of measures for use in survey questionnaires. Organizational Research Methods, 1, 104–121.

    Article  Google Scholar 

  • Hinkin, T. R., & Tracey, J. B. (1999). An analysis of variance approach to content validation. Organizational Research Methods, 2, 175–186.

    Article  Google Scholar 

  • Holden, R. R., & Jackson, D. N. (1979). Item subtlety and face validity in personality assessment. Journal of Consulting and Clinical Psychology, 47, 459.

    Article  Google Scholar 

  • Krosnick, J. A. (1999). Survey research. Annual Review of Psychology, 50, 537–567.

    Article  PubMed  Google Scholar 

  • Lawrence, S. A., Gardner, J., & Callan, V. J. (2007). The support appraisal for work stressors inventory: Construction and initial validation. Journal of Vocational Behavior, 70, 172–204.

    Article  Google Scholar 

  • Lawshe, C. H. (1975). A quantitative approach to content validity. Personnel Psychology, 28, 563–575.

    Article  Google Scholar 

  • Leach, C. W., van Zomeren, M., Zebel, S., Vliek, M. L. W., Pennekamp, S. F., Doosje, B., et al. (2008). Group-level self-definition and self-investment: A hierarchical (multicomponent) model of in-group identification. Journal of Personality and Social Psychology, 95, 144–165. doi:10.1037/0022-3514.95.1.144.

    Article  PubMed  Google Scholar 

  • Linderbaum, B. A., & Levy, P. E. (2010). The development and validation of the feedback orientation scale (FOS). Journal of Management, 36, 1372–1405.

    Article  Google Scholar 

  • MacCallum, R. C., Widaman, K. F., Zhang, S., & Hong, S. (1999). Sample size in factor analysis. Psychological Methods, 4, 84–99.

    Article  Google Scholar 

  • Mathwick, C., Malhotra, N., & Ridgon, E. (2001). Experiential value: Conceptualization, measurement and application in the catalog and internet shopping environment. Journal of Retailing, 77, 39–56.

    Article  Google Scholar 

  • Menor, L. J., & Roth, A. V. (2007). New service development competence in retail banking: Construct development and measurement validation. Journal of Operations Management, 25(4), 825–846.

    Article  Google Scholar 

  • Michel, J. S., Pace, V. L., Edun, A., Sawhney, E., & Thomas, J. (2014). Development and validation of an explicit aggressive beliefs and attitudes scale. Journal of Personality Assessment, 96, 327–338.

    Article  PubMed  Google Scholar 

  • Olson, K. (2010). An examination of questionnaire evaluation by expert reviewers. Field Methods, 22, 295–318.

    Article  Google Scholar 

  • Presser, S., & Blair, J. (1994). Survey pretesting: Do different methods produce different results? Sociological Methodology, 24, 73–104.

    Article  Google Scholar 

  • Presser, S., Couper, M. P., Lessler, J. T., Martin, E., Martin, J., Rothgeb, J. M., & Singer, E. (2004). Methods for testing and evaluating survey questions. Public Opinion Quarterly, 68, 109–130. doi:10.1093/poq/nfh008.

    Article  Google Scholar 

  • Rovinelli, R. J., & Hambleton, R. K. (1977). On the use of content specialists in the assessment of criterion-referenced test item validity. Dutch Journal of Educational Research, 2, 49–60.

    Google Scholar 

  • Schaeffer, N. C., & Presser, S. (2003). The science of asking questions. Annual Review of Sociology, 29, 65–88.

    Article  Google Scholar 

  • Spector, P. E. (1992). Summed rating scale construction. Newbury Park, CA: Sage Publications.

    Book  Google Scholar 

  • Stanton, J. M., Sinar, E. F., Balzer, W. K., & Smith, P. C. (2002). Issues and strategies for reducing the length of self-report scales. Personnel Psychology, 55, 167–194.

    Article  Google Scholar 

  • Tinsley, C. (1998). Models of conflict resolution in Japanese, German, and American cultures. Journal of Applied Psychology, 83, 316–323. doi:10.1037/0021-9010.83.2.316.

    Article  Google Scholar 

  • Ulaga, W., & Eggert, A. (2006). Value-based differentiation in business relationships: Gaining and sustaining key supplier status. Journal of Marketing, 70, 119–136.

    Article  Google Scholar 

  • Willis, G. B. (2005). Cognitive interviewing: A tool for improving questionnaire design. Thousand Oaks, CA: Sage.

    Google Scholar 

Download references

Acknowledgments

We would like to thank Matt Crayne for his comments on a previous version of this manuscript.

Author information

Authors and Affiliations

  1. Industrial/Organizational Psychology, The Pennsylvania State University, 142 Moore Building, University Park, PA, 16802, USA

    Matt C. Howard & Robert C. Melloy

Authors
  1. Matt C. Howard
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Robert C. Melloy
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Matt C. Howard.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Howard, M.C., Melloy, R.C. Evaluating Item-Sort Task Methods: The Presentation of a New Statistical Significance Formula and Methodological Best Practices. J Bus Psychol 31, 173–186 (2016). https://doi.org/10.1007/s10869-015-9404-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10869-015-9404-y

Keywords

Search

Navigation