Skip to main content
SpringerLink
Log in

An assessment of equivalence between Internet and paper-based surveys: evidence from collectivistic cultures

  • Published:
Quality & Quantity Aims and scope Submit manuscript

Abstract

Little research exists that addresses the equivalence in collectivistic cultures of paper- versus Internet-based surveys. This study addressed this gap and examined the measurement equivalence of individual innovativeness scales between Internet surveys and paper-based surveys within a collectivistic culture (with China serving as our example). The study analyzed and compared survey data from both paper and web-based surveys using confirmatory factor analysis. The assessment of invariance included the levels of configural, metric, scalar, and covariance invariance. The means and variance of latent variables were also compared. The results show that measurements are invariant at the two levels (configural and metric), and the covariances between latent variables are also equivalent, but the mean and variance differences of latent variables are apparent. The results indicate that when conducting research in collectivistic cultures and collecting data from distinct survey modes, researchers should concern themselves with the potential of extreme response patterns and the inclination of social desirability responding, as well as considering the measurement invariance across survey modes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Agarwal R., Karahanna E.: Time flies when you’re having fun: cognitive absorption and beliefs about information technology usage. MIS Q. 24(4), 665–694 (2000)

    Article  Google Scholar 

  • Agarwal R., Parsad J.: A conceptual and operational of personal innovativeness in the domain of information technology. Inf. Syst. Res. 9(3), 204–215 (1998)

    Article  Google Scholar 

  • Bao Y., Zhou K., Su C.: Face consciousness and risk aversion: do they affect consumer decision-making?. Psychol. Mark. 20(8), 733–755 (2003)

    Article  Google Scholar 

  • Barenboym D.A., Wurm L.H., Cano A.: A comparison of stimulus ratings made online and in person: gender and method effects. Behav. Res. Method 42(1), 273–285 (2010)

    Article  Google Scholar 

  • Bentler P.M.: EQS 6 Structural Equations Program Manual. Multivariate Software, Encino (2005)

    Google Scholar 

  • Booth-Kewley S., Larson G.E., Miyoshi D.K.: Social desirability effects on computerized and paper-and-pencil questionnaire. Comput. Hum. Behav. 23(1), 463–477 (2007)

    Article  Google Scholar 

  • Bosnjak M., Tuten T.L., Wittmann W.W.: Unit (non) response in web-based access panel surveys: an extended planned-behavior approach. Psychol. Mark. 22(6), 489–505 (2005)

    Article  Google Scholar 

  • Buchanan T., Johnson J.A., Goldberg L.R.: Implementing a five-factor personality inventory for use on the Internet. Eur. J. Psychol. Assess. 21(2), 116–128 (2005)

    Article  Google Scholar 

  • Chen F.: Sensitivity of goodness of fit indexes to lack of measurement invariance. Struct. Equ. Model. 14(3), 464–504 (2007)

    Article  Google Scholar 

  • Chiu Y., Lin C., Tang L.: Gender differs: assessing a model of online purchase intentions in E-tail service. Int. J. Serv. Ind. Manag. 16(5), 416–435 (2005)

    Article  Google Scholar 

  • Chou C.: Computer networks in communication survey research. IEEE Trans. Prof. Commun. 40(3), 196–208 (1997)

    Google Scholar 

  • Clark R., Goldsmith R.: Interpersonal influence and consumer innovativeness. Int. J. Consum. Stud. 30(1), 34–43 (2006)

    Article  Google Scholar 

  • Cole M., Bedeian A., Field H.: The measurement equivalence of web-based and paper-and-pencil measures of transformational leadership. Organ. Res. Methods 9(3), 339–368 (2006)

    Article  Google Scholar 

  • Davidov E., Depner F.: Testing for measurement equivalence of human values across online and paper-and-pencil surveys. Qual. Quant. 45(2), 375–390 (2011)

    Article  Google Scholar 

  • Davison R.M., Li Y., Kam C.S.P.: Web-based data collection in China. J. Glob. Inf. Manag. 14(3), 70–89 (2006)

    Article  Google Scholar 

  • Dwight S.A., Feigelson M.E.: A quantitative review of the effect of computerized testing on the measurement of social desirability. Educ. Psychol. Meas. 60(3), 340–360 (2000)

    Article  Google Scholar 

  • Fan W., Yan Z.: Factors affecting response rates of the web survey: a systematic review. Comput. Hum. Behav. 26(2), 132–139 (2010)

    Article  Google Scholar 

  • Fang J., Wen C., Pavur R.: Participation willingness in web surveys: exploring effect of sponsoring corporation’s and survey provider’s reputation. Cyberpsychol. Behav. Soc. Netw. 15(4), 195–199 (2012)

    Article  Google Scholar 

  • Fouladi R.T., McCarthy C.J., Moller N.P.: Paper-and-pencil or online? Evaluating mode effects on measures of emotional functioning and attachment. Assessment 9(2), 204–215 (2002)

    Article  Google Scholar 

  • French B.F., Finch W.H.: Multigroup confirmatory factor analysis: locating the invariant referent sets. Struct. Equ. Model. 15(1), 96–113 (2008)

    Article  Google Scholar 

  • Göritz A.S.: Incentives in web studies: methodological issues and a review. Int. J. Internet Sci. 1(1), 58–70 (2006)

    Google Scholar 

  • Ha Y., Stoel L.: Internet apparel shopping behaviors: the influence of general innovativeness. Int. J. Retail Distrib. Manag. 32(8), 377–385 (2004)

    Article  Google Scholar 

  • Hair J.F., Black W.C., Anderson R.E., Tatham R.L.: Multivariate Data Analysis, 6th edn.. Prentice-Hall, Upper Saddle River (1998)

    Google Scholar 

  • Heerwegh D., Loosveldt G.: Personalizing E-mail contacts: its influence on web survey response rate and social desirability response bias. Int. J. Public Opin. Res. 19(2), 258–268 (2007)

    Article  Google Scholar 

  • Higgins L., Zheng M., Liu Y., Sun C.: Attitudes to marriage and sexual behaviors: a survey of gender and culture differences in China and United Kingdom. Sex Roles 46(3/4), 75–89 (2002)

    Article  Google Scholar 

  • Hurt T., Joseph K., Cook C.: Scales for the measurement of innovativeness. Hum. Commun. Res. 4(1), 58–65 (1977)

    Article  Google Scholar 

  • Keh H.T., Sun J.: The complexities of perceived risk in cross-cultural services marketing. J. Int. Mark. 16(1), 120–146 (2008)

    Article  Google Scholar 

  • King W., Miles E.: Quasi-experimental assessment of the effect of computerizing noncognitive paper-and-pencil measurements: a test of measurement equivalence. J. Appl. Psychol. 80(6), 643–651 (1995)

    Article  Google Scholar 

  • Lalwani A., Chiu C.: Cognitive load, need for closure, and socially desirable responding: cognitively constrained versus motivated response biases in cross-cultural consumer research. Adv. Consum. Res. 35, 759–760 (2008)

    Google Scholar 

  • Leidner D.E., Kayworth T.: A review of culture in information systems research: toward a theory of information technology culture conflict. MIS Q. 30(2), 357–399 (2006)

    Google Scholar 

  • Lu J., Yao J.E., Yu C.S.: Personal innovativeness, social influences and adoptation of wireless Internet services via mobile technology. J. Strateg. Inf. Syst. 14(3), 245–268 (2005)

    Article  Google Scholar 

  • Mcknight D.H., Choudhury V., Kacmar C.: Developing and validating trust measures for E-commerce: an integrative typology. Inf. Syst. Res. 13(3), 334–359 (2002)

    Article  Google Scholar 

  • Meade A.W., Michels L.C., Lautenschlager G.J.: Are Internet and paper-and pencil personality tests truly comparable? An experimental design measurement invariance study. Organ. Res. Methods 10(2), 322–345 (2007)

    Article  Google Scholar 

  • Miles E.W., King W.C.: Gender and administration mode effects when pencil-and-paper personality tests are computerized. Educ. Psychol. Meas. 58(1), 68–76 (1998)

    Article  Google Scholar 

  • Nunnally J.C., Bernstein I.H.: Psychometric Theory, 3rd edn). McGraw-Hill, New York (1994)

    Google Scholar 

  • Paulhus D.L.: Two-component models of social desirable responding. J. Pers. Soc. Psychol. 46(3), 598–609 (1984)

    Article  Google Scholar 

  • Raju S., Laffitte J., Byrne M.: Measurement equivalence: a comparison of methods based on confirmatory factor analysis and item response theory. J. Appl. Psychol. 87(3), 517–529 (2002)

    Article  Google Scholar 

  • Reise S.P., Widaman K.F., Pugh R.H.: Confirmatory factor analysis and item response theory: two approaches for exploring measurement equivalence. Psychol. Bull. 114(3), 552–566 (1993)

    Article  Google Scholar 

  • Rosen, P.A.: The effect of personal innovativeness in the domain of information technology on the acceptance and use of technology. A Working Paper (2004)

  • Steinmetz H., Schmidt P., Tina-Booh A., Wieczorek S., Schwartz S.: Testing measurement invariance using multigroup CFA: differences between educational groups in human values measurement. Qual. Quant. 43(4), 599–616 (2009)

    Article  Google Scholar 

  • Tellis G., Chandrasekaran D.: Extent and impact of response biases in cross-national survey research. Int. J. Res. Mark. 27(4), 329–341 (2010)

    Article  Google Scholar 

  • Vandenberg R.J.: Toward a further understanding of and improvement in measurement invariance methods and procedure. Organ. Res. Methods 5(2), 139–158 (2002)

    Article  Google Scholar 

  • Vandenberg J., Lance E.: A review and synthesis of the measurement invariance literature: suggestions, practices, and recommendations for organizational research. Organ. Res. Methods 3(1), 4–70 (2002)

    Article  Google Scholar 

  • Vecchione M., Alessandri G., Barbaranelli C.: Paper-and-pencil and web-based testing: the measurement invariance of the big five personality tests in applied settings. Assessment 19(2), 243–246 (2012)

    Article  Google Scholar 

  • Westland J.C.: Lower bounds on sample size in structural equation modeling. Electron. Commer. Res. Appl. 9(6), 476–487 (2010)

    Article  Google Scholar 

  • Yetter G., Capaccioli K.: Differences in responses to web and paper surveys among school professionals. Behav. Res. Methds 42(1), 266–272 (2010)

    Article  Google Scholar 

  • Zhang N., Guo X., Chen G., Chau P.: Impact of perceived fit on E-government user evaluation: a study with a Chinese cultural context. J. Glob. Inf. Manag. 17(1), 49–69 (2009)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Management Science and E-Business, School of Management and Economics, University of Electronic Science and Technology of China, No. 4, Section 2, N. Jianshe Rd, Chengdu, 610054, Sichuan, China

    Jiaming Fang

  2. School of Business, Eastern Illinois University, 600 Lincoln Avenue, Charleston, IL, 61920, USA

    Chao Wen

  3. Department of Information Technology and Decision Sciences, College of Business, University of North Texas, 1155 Union Circle #305249, Denton, TX, 76203, USA

    Victor R. Prybutok

Authors
  1. Jiaming Fang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chao Wen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Victor R. Prybutok
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jiaming Fang.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Fang, J., Wen, C. & Prybutok, V.R. An assessment of equivalence between Internet and paper-based surveys: evidence from collectivistic cultures. Qual Quant 48, 493–506 (2014). https://doi.org/10.1007/s11135-012-9783-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11135-012-9783-3

Keywords

Search

Navigation