Information quality, user satisfaction, and the manifestation of workarounds: a qualitative and quantitative study of enterprise content management system users

Abstract

In this paper, we focus on a critical aspect of work in organizations: using information in work tasks which is provided by information systems (IS) such as enterprise content management (ECM) systems. Our study based on the IS success model, 34 interviews, and an empirical study of 247 ECM system users at a financial service provider indicates that it is appropriate to differentiate between contextual and representational information quality as two information quality dimensions. Furthermore, we reveal that in addition to system quality, the two information quality dimensions are important in determining end-user satisfaction, which in turn influences the manifestation of workarounds. Our study also finds that employees using workarounds to avoid an ECM system implemented several years is negatively related to individual net benefits of the ECM system. Hence, we conclude that when investigating large-scale IS such as ECM systems, it is important to differentiate among information quality dimensions to more deeply understand end-user satisfaction and the resulting manifestation of workarounds. Moreover, this research guides organizations in implementing the most appropriate countermeasures based on the importance of either contextual or representational information quality.

Appendices

Appendix A: Item development for workarounds

We develop and validate measurement items for workarounds in four steps in line with methods used in prior research developing new scales and developing new items systematically and rigorously (e.g., Chin et al, 1997; Maier et al, 2015a; Ragu-Nathan et al, 2008; Salisbury et al, 2002).

Step 1: Item development

In a first step, we scanned the recent literature discussing user resistance behavior (see Table A9 for an overview). In this step, we identified some user resistance behaviors related to IS implementations and developed an initial set of items for the proposed variable. We focused especially on those papers taking qualitative approaches to identify workarounds and used the example statements provided to define a first set of items for a variable that can capture the manifestation of workarounds in an empirical study.

In parallel, we also interviewed 20 employees in the target organization (see methodology section) to (a) identify examples of users’ workarounds in the organization and (b) identify the drivers and consequences of workaround behavior. Based on our analysis of the literature and the interviews, we developed a pool of items as illustrated in Table A1.

Table A1 Items of workarounds and q-sorting results (items below 0.61 are removed, here: WA-5)

Step 2: Assessing reliability and construct validity of the new items

Using q-sorting, we tested the reliability and validity of the new proposed items (Landis and Koch, 1977; Nahm et al, 2002) inviting 39 students from our university to participate in a q-sorting test. We developed a list of items including the newly developed ones for workarounds, items proposed by Kim & Kankanhalli (2009) and by Laumer et al (2014) for different user resistance behaviors. We included the existing ones as it is recommended by q-sorting to include items of similar constructs. Hence, items of two established user resistance constructs were included alongside the newly developed ones for workarounds. Moreover, we created an introductory statement defining each variable and instructions how to proceed. In the test, each individual read the introductory statement and assigned each item to one of the three constructs. Based on the assignment, we calculated ratios to evaluate the number of individuals matching the items to the correct variable. Using these results, we removed each item (WA-5) which was assigned by less than 61%, as suggested by prior research (Landis and Koch, 1977).

Step 3: Exploratory and confirmatory factor analysis

Using the remaining items of step 2, we conducted an additional survey in an organization focusing on another ECM system, as described above. The purpose of this survey was to collect data for an exploratory and confirmatory factor analysis to further evaluate the validity and reliability of the items for the new constructs. Therefore, a questionnaire was developed focusing on ECM usage, workarounds, user resistance, and several perceptions of the ECM in the organization. The data collected were used to conduct an exploratory factor analysis using SPSS 22. For this test, we used the newly developed items and the ones proposed for user resistance (Kim & Kankanhalli, 2009) and employee grumbling (Laumer et al, 2014). Our results reveal a three-factor structure. In a second step, the dataset was used to perform a confirmatory factor analysis using SPSS 23. Both steps revealed the same factor structured as illustrated in Table A2.

Table A2 Factor analysis

Table A3 q-sorting results

Step 4: Construct reliability and discriminant validity

In a next step, we focused on the reliability and discriminant validity of the newly proposed measurement model of workarounds, calculating Cronbach’s alpha for the remaining variables of step 3. The resulting value of 0.82 indicates a good construct reliability of the newly developed measurement model for workarounds (Hair, 2010; Nunnally, 1967). Furthermore, we performed again an explorative factor analysis to ensure convergent and discriminant validity. In this step, each item was assigned to the intended construct, confirming convergent and discriminant validity.

In summary, the measurement development process resulted in four items, which were used for the newly proposed variable of workarounds in validating the proposed research model.

Appendix B: Discriminant validity of contextual and representational information quality

To further test the discriminant validity of contextual and representational information quality, additional studies were conducted. The purpose of these studies was to collect data for both a q-sorting study and an exploratory and confirmatory factor analysis to further evaluate the validity and reliability of the two information quality dimensions.

Using q-sorting, we tested the reliability and validity of the proposed information quality dimensions (Landis & Koch, 1977; Nahm et al, 2002) inviting 28 students from our university to participate in a q-sorting test. We included the characteristics identified in our qualitative study. We created an introductory statement defining the two dimensions and instructions how to proceed. In the test, each individual read the introductory statement and assigned each characteristic to one of the two dimensions. Based on the assignment, we calculated ratios to evaluate the number of individuals matching the items to the correct dimensions. As no characteristic was assigned by less than 61%, we conclude that the assignment of the characteristics to the two information quality dimensions is reliable and valid as suggested by prior research (Landis & Koch, 1977).

Furthermore, we also ran a factor analysis with the data collected in the main study of this paper. Also these tests reveal a two-factor structure using the characteristics identified in our qualitative study. Consequently, we include this structure in the main study to analyze the effect of representational and contextual information quality on the manifestation of workarounds.

Appendix C: Measurement items

Table A4 summarizes the definition for each quality characteristic used in our survey instrument.

Table A4 Definition of quality characteristics

Table A5 illustrates the measurement items used and the respecting loadings of each item for the respective construct.

Table A5 Measurement items and loadings

Appendix D: Measurement model validation

Table A6 illustrates the reliability of the first-order constructs and the correlations between them.

Table A6 First-order reliability, AVEs, and correlation of constructs

Table A7 illustrates the cross-loadings of the items of the first-order constructs. The items are shown in Table A5 and are used in Table A7 in the same order as illustrated in Table A5.

Table A7 Cross-loadings (first-order constructs)

Appendix E: ECM system vendors

Table A8 provides an overview of ECM software vendors.

Table A8 ECM software vendor's overview based on Gartner’s magic ECM quadrant 2013 (Gilbert et al, 2013)

Appendix F: User resistance studies

Workaround is one example of user resistance behavior observed in relation to the usage of enterprise systems such as ECM. There are other forms of user resistance behaviors which have already been discussed by prior research (see Table A9), whereas only few articles have explicitly focused on workarounds. Studies focusing on workarounds as user resistance behavior have used interviews to identify and describe potential ways users can work around an IS. They have not provided and applied an instrument for further empirical analysis in this area.

Table A9 User resistance behavior