Skip to main content
SpringerLink
Log in

A multi-facet item response theory approach to improve customer satisfaction using online product ratings

  • Original Empirical Research
  • Published:
Journal of the Academy of Marketing Science Aims and scope Submit manuscript

Abstract

While online platforms often provide a single composite rating and the ratings of different attributes of a product, they largely ignore the attribute characteristics and customer criticality, which limits managerial action. We propose a multi-facet item response theory (MFIRT) approach to simultaneously examine the effects of product attributes, reviewer criticality, consumption situation, product type, and time in assessing latent customer satisfaction. Analyses of hotel ratings from TripAdvisor and beer ratings from BeerAdvocate suggest that product attributes differ with respect to their discriminating and threshold characteristics and that reviewer segments emphasize different attributes when rating various products over time. The MFIRT approach predicts product performance more accurately than alternative methods and provides novel insights to inform marketing strategies. The MFIRT framework can fundamentally advance how we analyze customer satisfaction and other consumer attitudes and improve marketing research and practice.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Notes

  1. A summary table is included in Web Appendix A for interested readers to compare IRT and CTT in terms of their assumptions, mathematical model, nature of variables, precision of meaurement, and strengths and weaknesses.

  2. Web Appendix D reports the details of multilevel GRM models. For each specified model, we run 5000 samples and discard them as burn-in. We use 20,000 iterations to generate the sample to estimate the parameters. We derive the attribute weights from the results of Model 2.

  3. The ML-MFIRT commands implemented using SAS NLMIXED and detailed results are reported in Web Appendix E.

  4. The reviewers of Tripadvisor are assigned a title (i.e., reviewers withouta title or those who are starting out as reviewers, senior reviewers, contributors, senior contributors, and top contributors) based on their increasing contributions.

  5. To save space, we do not report the results of response category parameters in Table 4. Interested readers can check the detailed results in Table E1 of Web Appendix E.

  6. The details of preprocessing and modeling in the text analysis of customer reviews are reported in Web Appendix F.

  7. The detailed output is presented in Table E2 of Web Appendix E.

References

  • Adams, R. J., Wilson, M., & Wu, M. (1997). Multilevel item response models: An approach to errors in variables regression. Journal of Educational and Behavioral Statistics, 22(1), 47–76.

    Article  Google Scholar 

  • Anderson, C. (2008). The Long Tail: Why the Future of Business is Selling Less of More. New York: Hyperion.

    Google Scholar 

  • Anderson, E. W., & Mittal, V. (2000). Strengthening the satisfaction-profit chain. Journal of Service Research, 3(2), 107–120.

    Article  Google Scholar 

  • Asparouhov, T., & Muthén, B. (2012). General random effect latent variable modeling: random subjects, items, contexts, and parameters. In J. R. Harring, L. M. Stapleton, & S. N. Beretvas (Eds.), Advances in Multilevel Modeling for Educational Research: Addressing Practical Issues Found in Real-world Applications (pp. 163–192). Charlotte: Information Age Publishing, Inc.

    Google Scholar 

  • Balasubramanian, S. K., & Kamakura, W. A. (1989). Measuring consumer attitudes toward the marketplace with tailored interviews. Journal of Marketing Research, 26(3), 311–328.

    Article  Google Scholar 

  • Balasubramanian, S. K., Konana, P., & Menon, N. M. (2003). Customer satisfaction in virtual environments: A study of online investing. Management Science, 49(7), 871–889.

    Article  Google Scholar 

  • Bawa, K., Srinivasan, S. S., & Srivastava, R. K. (1997). Coupon attractiveness and coupon proneness: A framework for modeling coupon redemption. Journal of Marketing Research, 34(4), 517–525.

    Article  Google Scholar 

  • Bechtel, G. G. (1985). Generalizing the rasch model for consumer rating scales. Marketing Science, 4(1), 62–73.

    Article  Google Scholar 

  • Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent dirichlet allocation. Journal of Machine Learning Research, 3, 993–1022.

    Google Scholar 

  • Büschken, J., & Allenby, G. M. (2016). Sentence-based text analysis for customer reviews. Marketing Science, 35(6), 953–975.

    Article  Google Scholar 

  • Cambria, E., & White, B. (2014). Jumping NLP curves: A review of natural language processing research. IEEE Computational Intelligence, 9(2), 48–57.

    Article  Google Scholar 

  • Carenini, G., Cheung, J. C., & Pauls, A. (2013). Multi-document summarization of evaluative text. Computational Intelligence, 29(4), 545–576.

    Article  Google Scholar 

  • Chen, P., Hong, Y., & Liu, Y. (2018). The value of multi-dimensional rating systems: Evidence from a natural experiment and randomized experiments. Management Science, 64(10), 4629–4647.

    Article  Google Scholar 

  • Chevalier, J., & Mayzlin, D. (2006). The effect of word of mouth on sales: Online book reviews. Journal of Marketing Research, 43(3), 345–354.

    Article  Google Scholar 

  • Chou, Y. T., & Wang, W. C. (2010). Checking dimensionality in item response models with principal component analysis on standardized residuals. Educational and Psychological Measurement, 70(5), 717–731.

    Article  Google Scholar 

  • Chung, W., & Tseng, B. (2012). Discovering business intelligence from online product reviews: A rule-induction framework. Expert Systems with Applications, 39, 11870–11879.

    Article  Google Scholar 

  • Clemons, E. K., Gao, G., & Hitt, L. M. (2006). When online reviews meet hyperdifferentiation: A study of the craft beer industry. Journal of Management Information Systems, 23(2), 149–171.

    Article  Google Scholar 

  • Cui, G., Lui, H. K., & Guo, X. (2012). The effect of online consumer reviews on new product sales. International Journal of Electronic Commerce, 17(1), 39–57.

    Article  Google Scholar 

  • de Jong, M., Steenkamp, J. E. M., & Fox, J. (2007). Relaxing measurement invariance in cross-national consumer research using a hierarchical IRT model. Journal of Consumer Research, 34(2), 260–279.

    Article  Google Scholar 

  • de Jong, M., Steenkamp, J. E. M., Fox, J., & Baumgartner, H. (2008). Using item response theory to measure extreme response style in marketing research: A global investigation. Journal of Marketing Research, 45(1), 104–115.

    Article  Google Scholar 

  • de Jong, M., Steenkamp, J. E. M., & Veldkamp, B. P. (2009). A model for the construction of country-specific yet internationally comparable short-form marketing scales. Marketing Science, 28(4), 674–689.

    Article  Google Scholar 

  • de Jong, M., Pieters, R., & Fox, J. (2010). Reducing social desirability bias through item randomized response: An application to measure underreported desires. Journal of Marketing Research, 47(1), 14–27.

    Article  Google Scholar 

  • De Langhe, B., Fernbach, P. M., & Lichtenstein, D. R. (2016). Navigatig by the stars: Investigating the actual and perceived validity of online user ratings. Journal of Consumer Research, 42(6), 817–833.

    Article  Google Scholar 

  • Dellarocas, C. (2003). The digitization of word of mouth: Promise and challenges of online feedback mechanisms. Management Science, 49(10), 1407–1424.

    Article  Google Scholar 

  • Dellarocas, C., Zhang, M. X., & Awad, N. F. (2007). Exploring the value of online product reviews in forecasting sales: The case of motion pictures. Journal of Interactive Marketing, 21(4), 23–45.

    Article  Google Scholar 

  • DeSarbo, W. S., Wedel, M., Vriens, M., & Ramaswamy, V. (1992). Latent class metric conjoint analysis. Marketing Letters, 3(3), 273–288.

  • Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists, Lawrence.

    Google Scholar 

  • Ewing, M. T., Salzberger, T., & Sinkovics, R. R. (2005). An alternate approach to assessing cross-cultural measurement equivalence in advertising research. Journal of Advertising Research, 34(1), 17–36.

    Article  Google Scholar 

  • Finn, A., & Kayande, U. (1997). Reliability assessment and optimization of marketing measurement. Journal of Marketing Research, 22(1), 262–275.

    Article  Google Scholar 

  • Finn, A., & Kayande, U. (2005). How fine is COARSE? A generalizability theory perspective on rossiter’s procedure. International Journal of Research in Marketing, 22(1), 11–21.

    Article  Google Scholar 

  • Fishbein, M. (1983). An investigation of the relationships between beliefs about an object and the attitude toward that objec. Human Relations, 16(3), 233–240.

    Article  Google Scholar 

  • Fox, J. (2005). Multilevel IRT using dichotomous and polytomous response data. British Journal of Mathematical and Statistical Psychology, 58(1), 145–172.

    Article  Google Scholar 

  • Fox, J., & Glas, C. A. W. (2001). Bayesian estimation of a multilevel IRT model using Gibbs sampling. Psychometrika, 66, 269–286.

    Article  Google Scholar 

  • Garbarino, E., & Johnson, M. S. (1999). The different roles of satisfaction, trust, and commitment in customer relationships. Journal of Marketing, 63(2), 70–87.

    Article  Google Scholar 

  • Gierl, M. J., & Khaliq, S. N. (2001). Identifying sources of differential item and bundle functioning on translated achievement tests. Journal of Educational Measurement, 38(2), 164–187.

    Article  Google Scholar 

  • Hambleton, R. K., Swaminathan, H., & Rogers, J. H. (1991). Fundamentals of item response theory. Newbury Park, CA: Sage.

    Google Scholar 

  • Hanson, R. (1992). Determining attribute importance. Quirk’s Marketing Research Review, 6(October), 16–18.

    Google Scholar 

  • Hennig-Thurau, T., Gwinner, K. P., Walsh, G., & Gremler, D. D. (2004). Electronic word of mouth via consumer-opinion platforms: What motivates consumers to articulate themselves on the internet. Journal of Interactive Marketing, 18(1), 38–52.

    Article  Google Scholar 

  • Holbrook, M. (2005). The role of ordinary evaluations in the market for popular culture: Do consumers have good taste. Marketing Letters, 16(2), 75–86.

    Article  Google Scholar 

  • Holman, R., Glas, C. A. W., & de Haan, R. J. (2003). Power analysis in randomized clinical trials based on item response theory. Controlled Clinical Trials, 24(4), 390–410.

    Article  Google Scholar 

  • Hung, L. F., & Wang, W. C. (2012). The generalized multilevel facets model for longitudinal data. Journal of Educational and Behavioral Statistics, 37(2), 231–255.

    Article  Google Scholar 

  • Kekre, S., Krishnan, M. S., & Srinivasan, K. (1995). Drivers of customer satisfaction for software products: Implications for design and support service. Management Science, 41(9), 1456–1470.

    Article  Google Scholar 

  • Knoll, M. A. Z., & Houts, C. R. (2012). The financial knowledge scale: An application of item response theory to the assessment of financial literacy. Journal of Consumer Affairs, 46(3), 381–410.

    Article  Google Scholar 

  • Koren, Y., Bell, R., & Volinsky, C. (2009). Matrix factorization techniques for recommender systems. Computer, 42(8), 30–37.

    Article  Google Scholar 

  • Lee, K., Lee, B., & Oh, W. (2015). Thumbs up, sales up? The contingent effect of facebook likes on sales performance in social commerce. Journal of Management Information Systems, 32(4), 109–143.

    Article  Google Scholar 

  • Li, X., & Hitt, L. M. (2008). Self-selection and information role of online product reviews. Information Systems Research, 19(4), 456–474.

    Article  Google Scholar 

  • Li, C., Peng, L., & Cui, G. (2017). Picking winners: New product concept testing with the item response theory. International Journal of Market Research, 59(3), 335–354.

    Article  Google Scholar 

  • Linacre, J. M. (1994). Many-facet Rasch measurement. Chicago: MESA Press.

    Google Scholar 

  • Lord, F. M. (1980). Applications of item response theory to practical testing problems. In Erlbaum. NJ: Hillside.

    Google Scholar 

  • Mattila, A. S. (1999). The role of culture in the service evaluation process. Journal of Service Research, 1(3), 250–261.

    Article  Google Scholar 

  • Maydeu-Olivares, A. (2013). Goodness-of-fit assessment of item response theory models. Measurement: Interdisciplinary Research and Perspectives, 11(3), 71–101.

    Google Scholar 

  • Mittal, V., & Kamakura, W. (2001). Satisfaction, repurchase intention, and repurchase behavior: Investigating the moderating effect of customer characteristics. Journal of Marketing Research, 38(1), 131–142.

    Article  Google Scholar 

  • Mittal, V., Ross, W. T., & Baldasare, P. M. (1998). The asymmetric impact of negative and positive attribute-level performance on overall satisfaction and repurchase intentions. Journa of Marketing, 62(1), 33–47.

    Article  Google Scholar 

  • Moe, W. W., & Schweidel, D. A. (2011). Online product opinion: Incidence, evaluations and evolution. Marketing Science, 31(3), 372–386.

    Article  Google Scholar 

  • Moe, W. W., & Trusov, M. (2011). The value of social dynamics in online product ratings forums. Journal of Marketing Research, 48(3), 444–456.

    Article  Google Scholar 

  • Mokkan, R. J., & Lewis, C. (1982). A nonparametric approach to the analysis of dichotomous item responses. Applied Psychological Measurement, 6(4), 417–430.

    Article  Google Scholar 

  • Netzer, O., Feldman, R., Goldenberg, J., & Fresko, M. (2012). Mine your own business: Market-structure surveillance through text mining. Marketing Science, 31(3), 521–543.

    Article  Google Scholar 

  • Ostini, R., & Nering, M. L. (2006). Polytomous item response theory models. Thousand Oaks: Sage.

    Book  Google Scholar 

  • Paas, L., & Sijtsma, K. (2008). Nonparametric item response theory for investigating dimensionality of marketing scales: A SERVQUAL application. Marketing Letters, 19(2), 157–170.

    Article  Google Scholar 

  • Raykov, T., & Calantone, R. J. (2014). The utility of item response modeling in marketing research. Journal of the Academy of Marketing Science, 42(4), 337–360.

    Article  Google Scholar 

  • Reise, S. P., & Waller, N. G. (2009). Item response theory and clinical measurement. Annual Review of Clinical Psychology, 5, 27–48.

    Article  Google Scholar 

  • Rusch, T., Lowry, P., Mair, P., & Treiblmaier, H. (2017). Breaking free from the limitations of classical test theory: Developing and measuring information systems scales using item response theory. Information & Management, 54, 189–203.

    Article  Google Scholar 

  • Samejima, F. (1969). Estimation of a latent ability using a response pattern of graded scores. Psychometrika Monograph Supplement, 34(4), 100–139.

    Google Scholar 

  • Schultz, C., Salomo, S., & Talke, K. (2013). Measuring new product portfolio innovativeness: How differences in scale width and evaluator perspectives affect its relationship with performance. Journal of Product Innovation Management, 30(S1), 93–109.

    Article  Google Scholar 

  • Singh, J., Howell, R. D., & Rhoads, G. K. (1990). Adaptive designs for likert-type data: An approach for implementing marketing surveys. Journal of Marketing Research, 27(3), 304–321.

    Article  Google Scholar 

  • Smith, D., Menon, S., & Sivakumar, K. (2005). Online peer and editorial recommendations, trust, and choice in virtual markets. Journal of Interactive Marketing, 19(3), 15–37.

    Article  Google Scholar 

  • Srivastava, A., & Sahami, M. (2009). Text mining: Classification, clustering, and Appications, Chapman and Hall.

  • Swaminathan, S., & Bawa, K. (2005). Category-specific coupon proneness: The impact of individual characteristics and category-specific variables. Journal of Retailing, 81(3), 205–214.

    Article  Google Scholar 

  • Tay, L., Diener, E., Drasgow, F., & Vermunt, J. K. (2011). Multilevel mixed-measurement IRT analysis: An explication and application to self-reported emotions across the world. Organizational Research Methods, 14(1), 177–207.

    Article  Google Scholar 

  • Van den Bulte, C., & Joshi, Y. V. (2007). New product diffusion with influentials and imitators. Marketing Science, 26(3), 400–421.

    Article  Google Scholar 

  • Van den Noortgate, W., De Boeck, P., & Meulders, M. (2003). Cross-classification multilevel logistic models in psychometrics. Journal of Educational and Behavioral Statistics, 28, 369–386.

    Article  Google Scholar 

  • Wang, W. C., & Liu, C. Y. (2007). Formulation and application of the generalized multilevel facets model. Educational and Psychological Measurement, 67(4), 583–605.

    Article  Google Scholar 

  • Xie, K. L., Zhang, Z., & Zhang, Z. (2014). The business value of online consumer reviews and management response to hotel performance. International Journal of Hospitality Management, 43, 1–12.

    Article  Google Scholar 

  • Xu, X., & Li, Y. (2016). The antecedents of customer satisfaction and dissatisfaction toward various types of hotels: A text mining approach. International Journal of Hospitality Management, 55, 57–69.

    Article  Google Scholar 

  • Zwass, V. (2010). Co-creation: Toward a taxonomy and an integrated research perspective. International Journal of Electronic Commerce, 15(1), 11–48.

    Article  Google Scholar 

Download references

Acknowledgements

The authors thank STR for providing the hotel performance data. This research was enabled by a General Research Fund from the Research Grants Council Hong Kong (Grant No. 13500314).

Author information

Authors and Affiliations

  1. Department of Marketing and International Business, Lingnan University, 8 Castle Peak Road, Tuen Mun, New Territories, Hong Kong

    Ling Peng, Geng Cui & Yuho Chung

  2. Department of Marketing, School of Business, Guangdong University of Foreign Studies, 2 North Baiyun Road, Guangzhou, China

    Chunyu Li

Authors
  1. Ling Peng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Geng Cui
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yuho Chung
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Chunyu Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ling Peng.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rajkumar Venkatesan served as Area Editor for this article.

Electronic supplementary material

ESM 1

(DOCX 137 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Peng, L., Cui, G., Chung, Y. et al. A multi-facet item response theory approach to improve customer satisfaction using online product ratings. J. of the Acad. Mark. Sci. 47, 960–976 (2019). https://doi.org/10.1007/s11747-019-00662-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11747-019-00662-w

Keywords

Search

Navigation