European Journal of Information Systems

, Volume 17, Issue 3, pp 236–263

Measuring information systems success: models, dimensions, measures, and interrelationships

Original Article

DOI: 10.1057/ejis.2008.15

Cite this article as:
Petter, S., DeLone, W. & McLean, E. Eur J Inf Syst (2008) 17: 236. doi:10.1057/ejis.2008.15

Abstract

Since DeLone and McLean (D&M) developed their model of IS success, there has been much research on the topic of success as well as extensions and tests of their model. Using the technique of a qualitative literature review, this research reviews 180 papers found in the academic literature for the period 1992–2007 dealing with some aspect of IS success. Using the six dimensions of the D&M model – system quality, information quality, service quality, use, user satisfaction, and net benefits – 90 empirical studies were examined and the results summarized. Measures for the six success constructs are described and 15 pairwise associations between the success constructs are analyzed. This work builds on the prior research related to IS success by summarizing the measures applied to the evaluation of IS success and by examining the relationships that comprise the D&M IS success model in both individual and organizational contexts.

Keywords

information systems successorganizational and individual use of ISIS effectivenessIT performancemeasurementliterature review

Introduction

In 2008, organizations continue to increase spending on information technology (IT) and their budgets continue to rise, even in the face of potential economic downturns (Kanaracus, 2008). However, fears about economic conditions and increasing competition create pressures to cut costs, which require organizations to measure and examine the benefits and costs of technology. Naturally, organizations are interested in knowing the return on these investments. The impacts of IT are often indirect and influenced by human, organizational, and environmental factors; therefore, measurement of information systems (IS) success is both complex and illusive.

IS are developed using IT to aid an individual in performing a task. Given the relatively short life-span of the IS field, it is quite remarkable the number and variety of applications and systems that have been deployed. There are IS that range from hedonic, developed for pleasure and enjoyment, to utilitarian, developed to improve individual and organizational performance (van der Heijden, 2004). Organizations focus on developing, using, and evaluating utilitarian IS. There is a plethora of utilitarian IS used in organizations, such as decision support systems, computer-mediated communications, e-commerce, knowledge management systems, as well as many others.

To measure the success of these various IS, organizations are moving beyond traditional financial measures, such as return on investment (Rubin, 2004). In an effort to better understand the tangible and intangible benefits of their IS, organizations have turned to methods such as balanced scorecards (Kaplan & Norton, 1996) and benchmarking (Seddon et al., 2002). Researchers have created models for success (DeLone & McLean, 1992; Ballantine et al., 1996; Seddon, 1997), emphasizing the need for better and more consistent success metrics.

As a field, we have made substantial strides towards understanding the nature of IS success. For example, the widely cited DeLone and McLean (D&M) model of IS success (1992) was updated a decade later based on a review of the empirical and conceptual literature on IS success that was published during this period (DeLone & McLean, 2003). Furthermore, some researchers have synthesized the literature by examining one or more of the relationships in the D&M IS success model using the quantitative technique of meta-analysis (Mahmood et al., 2001; Bokhari, 2005; Sabherwal et al., 2006) to develop a better understanding of success. Others have started to develop standardized measures that can be used to evaluate the various dimensions of IS success as specified by D&M (e.g., Sedera et al., 2004). This past research has helped the field better understand IS success, but more needs to be done. Therefore, this paper examines the research related to IS success to determine what is known and what still needs to be learned for utilitarian IS in an organizational context.

This qualitative literature review identified three objectives for improving the current understanding of the literature in this domain. First, the D&M model was examined in two different contexts: the individual level of analysis and the organizational level of analysis in order to identify if the unit of analysis under study is a boundary condition for measuring success. Second, unlike other literature reviews or meta-analyses that have only reviewed some of the relationships in the original D&M model, this review investigated all relationships in the updated IS success model (DeLone & McLean, 2003). Finally, the specific measures used by researchers for each of the constructs that comprise the D&M model were examined. In both the original and updated models, D&M strongly advocated the need for consistent and appropriate measures for IS success. This review seeks to determine whether researchers have heeded this call.

The next section explores various competing models of IS success and explains why the D&M model was chosen as the organizing framework for this literature review. A description of the methods used to obtain and classify the research, as well as a more detailed description of the literature review, is also provided. The results of this literature review are then presented, organized by the constructs contained in the D&M IS success model as well as the relationships between the constructs. The final section outlines the implications of this body of research for both practitioners and researchers interested in assessing information system success.

Background

The development of a model of IS success

Researchers have derived a number of models to explain what makes some IS ‘successful.’ Davis's (1989) Technology Acceptance Model (TAM) used the Theory of Reasoned Action and Theory of Planned Behavior (Fishbein & Ajzen, 1975) to explain why some IS are more readily accepted by users than others. Acceptance, however, is not equivalent to success, although acceptance of an information system is a necessary precondition to success.

Early attempts to define information system success were ill-defined due to the complex, interdependent, and multi-dimensional nature of IS success. To address this problem, DeLone & McLean (1992) performed a review of the research published during the period 1981–1987, and created a taxonomy of IS success based upon this review. In their 1992 paper, they identified six variables or components of IS success: system quality, information quality, use, user satisfaction, individual impact, and organizational impact. However, these six variables are not independent success measures, but are interdependent variables. Figure 1 shows this original IS success model (DeLone & McLean, 1992).
https://static-content.springer.com/image/art%3A10.1057%2Fejis.2008.15/MediaObjects/41303_2008_Article_BFejis200815_Fig1_HTML.gif
Figure 1

DeLone and McLean IS success model (1992).

Shortly after the publication of the D&M success model, IS researchers began proposing modifications to this model. Accepting the authors' call for ‘further development and validation,’ Seddon & Kiew (1996) studied a portion of the IS success model (i.e., system quality, information quality, use, and user satisfaction). In their evaluation, they modified the construct, use, because they ‘conjectured that the underlying success construct that researchers have been trying to tap is Usefulness, not Use’ (p. 93). Seddon and Kiew's concept of usefulness is equivalent to the idea of perceived usefulness in TAM by Davis (1989). They argued that, for voluntary systems, use is an appropriate measure; however, if system use is mandatory, usefulness is a better measure of IS success than use. DeLone & McLean (2003) responded that, even in mandatory systems, there can still be considerable variability of use and therefore the variable use deserves to be retained.

Researchers have also suggested that service quality be added to the D&M model. An instrument from the marketing literature, SERVQUAL, has become salient within the IS success literature within the past decade. SERVQUAL measures the service quality of IT departments, as opposed to individual IT applications, by measuring and comparing user expectations and their perceptions of the IT department. Pitt et al. (1995) evaluated the instrument from an IS perspective and suggested that the construct of service quality be added to the D&M model. Some researchers have resisted this change (Seddon, 1997), while others have endorsed it (Jiang et al., 2002). DeLone & McLean (2003), after reviewing and evaluating this debate, decided to add service quality in their updated IS success model stating that ‘the changes in the role of IS over the last decade argue for a separate variable – the “service quality” dimension’ (p. 18).

Another well-known proposed modification to the D&M model is the changes offered by Seddon (1997). He argued that the D&M model in its original form was confusing, partly because both process and variance models were combined within the same framework. While he claimed that this was a shortcoming of the model, DeLone & McLean (2003) responded that they believed that this was one of its strengths, with the insights provided, respectively, by process and variance models being richer than either is alone. Seddon furthered suggested that the concept of use is highly ambiguous and suggested that further clarification was needed to this construct. He derived three different potential meanings for the use construct, as well as parsing out the process and variances portions of the model. The D&M model of IS success was intended to be ‘both complete and parsimonious’; however, the changes introduced by Seddon complicates the model, thereby reducing its impact.

In addition to the modifications proposed by Seddon, there have been other calls to revise or extend the model. Some researchers have modified it to evaluate success of specific applications such as knowledge management (e.g., Jennex & Olfman, 2002; Kulkarni et al., 2006; Wu & Wang, 2006) and e-commerce (e.g., Molla & Licker, 2001; DeLone & McLean, 2004; Zhu & Kraemer, 2005). Other researchers have made more general recommendations concerning the model (e.g., Ballantine et al., 1996).

Recognizing these proposed modifications to their model, D&M, in a follow-up work, reviewed empirical studies that had been performed during the years since 1992 and revised the original model accordingly (DeLone & McLean, 2002, 2003). The updated model is shown in Figure 2.
https://static-content.springer.com/image/art%3A10.1057%2Fejis.2008.15/MediaObjects/41303_2008_Article_BFejis200815_Fig2_HTML.gif
Figure 2

Updated DeLone and McLean IS success model (2003).

This updated IS success model accepted the Pitt et al. (1995) recommendation to include service quality as a construct. Another update to the model addressed the criticism that an information system can affect levels other than individual and organizational levels. Because IS success affects workgroups, industries, and even societies (Myers et al., 1997; Seddon et al., 1999), D&M replaced the variables, individual impact and organizational impact, with net benefits, thereby accounting for benefits at multiple levels of analysis. This revision allowed the model to be applied to whatever level of analysis the researcher considers most relevant.

A final enhancement made to the updated D&M model was a further clarification of the use construct. The authors explained the construct as follows: ‘Use must precede “user satisfaction” in a process sense, but positive experience with “use” will lead to greater “user satisfaction” in a causal sense’ (DeLone & McLean, 2003). They went on to state that increased user satisfaction will lead to a higher intention to use, which will subsequently affect use.

The D&M model has also been found to be a useful framework for organizing IS success measurements. The model has been widely used by IS researchers for understanding and measuring the dimensions of IS success. Furthermore, each of the variables describing success of an information system was consistent with one or more of the six major success dimensions of the updated model. The dimensions of success include:
  • System quality – the desirable characteristics of an information system. For example: ease of use, system flexibility, system reliability, and ease of learning, as well as system features of intuitiveness, sophistication, flexibility, and response times.

  • Information quality – the desirable characteristics of the system outputs; that is, management reports and Web pages. For example: relevance, understandability, accuracy, conciseness, completeness, understandability, currency, timeliness, and usability.

  • Service quality – the quality of the support that system users receive from the IS department and IT support personnel. For example: responsiveness, accuracy, reliability, technical competence, and empathy of the personnel staff. SERVQUAL, adapted from the field of marketing, is a popular instrument for measuring IS service quality (Pitt et al., 1995).

  • System use – the degree and manner in which staff and customers utilize the capabilities of an information system. For example: amount of use, frequency of use, nature of use, appropriateness of use, extent of use, and purpose of use.

  • User satisfaction – users' level of satisfaction with reports, Web sites, and support services. For example, the most widely used multi-attribute instrument for measuring user information satisfaction can be found in Ives et al. (1983).

  • Net benefits – the extent to which IS are contributing to the success of individuals, groups, organizations, industries, and nations. For example: improved decision-making, improved productivity, increased sales, cost reductions, improved profits, market efficiency, consumer welfare, creation of jobs, and economic development. Brynjolfsson et al. (2002) have used production economics to measure the positive impact of IT investments on firm-level productivity.

The practical application of the D&M model is naturally dependent on the organizational context. The researcher wanting to apply the D&M model must have an understanding of the information system and organization under study. This will determine the types of measures used for each success dimension. The selection of success dimensions and specific metrics depend on the nature and purpose of the system(s) being evaluated. For example, an e-commerce application would have some similar success measures and some different success measures compared to an enterprise system application. Both systems would measure information accuracy, while only the e-commerce system would measure personalization of information. An information system that is managed by a vendor will measure the service quality of the vendor, rather than of the IS department. Seddon et al. (1999) developed a context matrix that is a valuable reference for the selection of success measures based on stakeholders and level of analysis (individual application or IS function). Ideally, the D&M model is applicable in a variety of contexts; however, the limits of the model are not well-known or understood. This research examines one of the potential boundary conditions for the model and identifies areas that warrant additional attention.

The current understanding of IS success

There have been a number of studies that have attempted to further the understanding of the D&M model by attempting to validate some, or all, of the entire model in a single study. Seddon & Kiew (1996) examined the relationships among four of the constructs and found good support. Rai et al. (2002) compared the original D&M model (1992) to the respecified model created by Seddon (1997) and found that the D&M model stood up reasonably well to the validation attempt and outperformed the Seddon model. Sedera et al. (2004) also recently tested several success models, including the D&M and Seddon models, against empirical data and determined that the DeLone & McLean Model provided the best fit for measuring enterprise systems success. McGill et al. (2003) examined the full model, but found four paths in the original IS success model insignificant (system quality → use, information quality → use, intended use → individual impact, and individual impact → organizational impact).

Some researchers (e.g., Au et al., 2002; DeLone & McLean, 2003; Grover et al., 2003) have conducted literature reviews to examine if the results of empirical studies support the relationships posited by the original success model. These literature reviews reveal that some relationships within the model have received consistent support (i.e., significant results across all studies) while others have received only mixed support (i.e., some studies find significant results while others are non-significant). Other researchers have performed meta-analyses to examine one or more relationships in the D&M model (e.g., Mahmood et al., 2001; Bokhari, 2005; Sabherwal et al., 2006). The most comprehensive meta-analysis examining the D&M model was performed by Sabherwal et al. (2006). Sabherwal et al.'s work has been instrumental in synthesizing the quantitative research related to IS success and has validated a substantial portion of the D&M model.

This study extends Sabherwal et al.'s work in several ways. First, by performing a qualitative literature review, studies that use qualitative methods or do not report enough information to be included in a meta-analysis can be included in this analysis of the literature. Second, by not aggregating each study into a single numerical value, it is possible to examine issues associated with measuring the various constructs within the D&M model and examine the lack of consistency among studies in terms of context and constructs. Third, this work examines if the level of analysis under study (i.e., the individual or the organization) is a boundary condition for the D&M model.

Methodology

One of the most well-established methods to integrate research findings and assess the cumulative knowledge within a domain is a qualitative literature review (Oliver, 1987). This method allows a researcher to analyze and evaluate both quantitative and qualitative literature within a domain to draw conclusions about the state of the field. As with any research technique, there are limitations. The primary limitation with this approach is that when conflicting findings arise, it becomes difficult to determine the reason for the conflicting results. Some also perceive that because the literature review is qualitative, it is subjective in nature and provides little ‘hard evidence’ to support a finding.

To counter these shortcomings, the research technique of meta-analysis has become quite popular in the social sciences and now in IS. Meta-analysis is an interesting and useful technique to synthesize the literature using quantitative data reported across research studies. The result of a meta-analysis is an ‘effect size’ statistic that states the magnitude of the relationship and whether or not the relationship between variables is statistically significant (Oliver, 1987; Hwang et al., 2000). This approach too has its limitations. A key limitation is the need to exclude studies that use qualitative techniques to examine success or studies that fail to report the information required for the statistical calculations for the meta-analysis. While the meta-analysis produces a quantified result regarding the relationship between two variables, the need to exclude some studies may not present a complete picture of the literature. Furthermore, a meta-analysis does not examine the direction of causality, because the effect size is an adjusted correlation between two variables.

There have been meta-analyses examining one or more of the elements of IS success (Hwang et al., 2000; Mahmood et al., 2001; Bokhari, 2005; Sabherwal et al., 2006); therefore, this paper seeks to obtain a different, qualitative view of the literature to answer a different set of research questions. While a meta-analysis is aimed at answering the question: ‘Is there a correlation between two variables?’, a qualitative literature review is better equipped to explain how the relationships have been studied in the literature, if there appears to be support for a causal relationship between two variables, and examines if there are any potential boundary conditions for the model.

Scope of the literature search

To find research that has been published on IS success, full-text searches in numerous online databases (EBSCO Host, ABI Inform, and Web of Knowledge) were performed using multiple keywords, such as ‘IS success,’ ‘IS effectiveness,’ ‘DeLone and McLean,’ etc. Print issues of well-known IS journals unavailable electronically were also examined to ensure that applicable studies were included.

As a means to ensure that the bibliography of relevant studies was complete, the list of studies was triangulated with the reference lists of several papers and Web sites that examined the history of IS success, such as the updated DeLone & McLean paper (2003) and an AISWorld Web site devoted to IS effectiveness (Grover et al., 2003). A total of 180 empirical and conceptual papers were identified in this wide-ranging search for IS success research (see Appendix A for a list of studies examined). These papers were published in the time period between 1992, the year the D&M success model was first published, and 2007. From this collection of papers, only papers reporting empirical results (both quantitative and qualitative research) of interrelationships among the D&M success dimensions are included in this paper; this yielded a total of 90 papers.

To perform a literature review, it is necessary to examine as much related literature as possible on the topic; however, to prevent from being overwhelmed, the focus of this research is on utilitarian IS that can be used by organizations or individuals to improve performance. Also, given that many other reference disciplines also study IS (e.g., marketing, psychology, management, etc.), the primary searches for literature focused on journals within the IS discipline. However, we did not restrict the literature review to a specific type of information system or a specific use context (i.e., individual vs organizational or voluntary vs mandatory).

Organizing the literature review

Because of the popularity of the D&M model in the academic literature, it seemed appropriate to organize the studies of IS success that were found using its taxonomy. The findings from empirical studies on IS success are organized by success constructs or dimensions. Subsections are grouped based on the expected causal relationships between paired success constructs. Organizing the literature in this manner helps to examine whether there is support for each of the proposed relationships within the D&M model (Figure 2). Table 1 lists each of the 15 causal relationships evaluated in this manuscript.
Table 1

Proposed success relationships posited in D&M model (2003)

System quality

System usea

System quality

User satisfaction

System quality

Net benefits

Information quality

System use

Information quality

User satisfaction

Information quality

Net benefits

Service quality

System use

Service quality

User satisfaction

Service quality

Net benefits

System use

User satisfaction

System use

Net benefits

User satisfaction

System use

User satisfaction

Net benefits

Net benefits

System use

Net benefits

User satisfaction

aWe chose to consider both intention to use and other measures of system use as the same construct for this literature review. Although D&M did distinguish between intention to use and system use in their updated model, intention to use is generally an individual level construct. This is not a concept that is consistent with studies employing an organizational unit of analysis. Furthermore, by parsing the use constructs into two separate subconstructs (i.e., intention to use and use), it makes an already complex paper (with 15 pairwise relationships) even more complex (adding at least six pairwise relationships to the analysis). In the discussion of the results of the literature review, we do identify those studies that measure intention to use as opposed to other measures of system use.

The subsections are further organized by the unit of analysis, whether individual or organizational stakeholders are the focus of the research. The updated D&M model suggests that IS success can be examined at different levels (DeLone & McLean, 2002, 2003); therefore, this literature review investigated if there are differences in the strengths of the relationships based on whether the studies focused on an individual or organizational level when measuring and evaluating the various success constructs and relationships.

Each of the constructs of the D&M model has multiple operationalizations; and the support, or lack of support, for relationships between constructs may be due to the manner in which the constructs were measured. Therefore, this review also discusses the specific success measures that were used in the selected studies.

Literature review of IS success studies

Measuring the six constructs of IS success

There are many approaches to measuring success of IS. Some researchers have developed approaches to measuring success in specific industries by incorporating the various dimensions of the D&M model (Weill & Vitale, 1999; Skok et al., 2001). However, there are many scales that have been used to measure the dimensions of the IS success model individually, with some being more thorough than others. This section identifies some of the different operationalizations of each construct.

Measuring system quality

Perceived ease of use is the most common measure of system quality because of the large amount of research relating to the TAM (Davis, 1989).

However, perceived ease of use does not capture the system quality construct as a whole. Rivard et al. (1997) developed and tested an instrument that consists of 40 items that measure eight system quality factors: namely, reliability, portability, user friendliness, understandability, effectiveness, maintainability, economy, and verifiability. Others have created their own indexes of system quality using the dimensions identified by D&M in their original model (Coombs et al., 2001) or via their own review of the system quality literature (Gable et al., 2003).

Measuring information quality

Information quality is often a key dimension of end-user satisfaction instruments (Ives et al., 1983; Baroudi & Orlikowski, 1988; Doll et al., 1994). As a result, information quality is often not distinguished as a unique construct but is measured as a component of user satisfaction. Therefore, measures of this dimension are problematic for IS success studies. Fraser & Salter (1995) developed a generic scale of information quality, and others have developed their own scales using the literature that is relevant to the type of information system under study (Coombs et al., 2001; Wixom & Watson, 2001; Gable et al., 2003).

Measuring service quality

As discussed earlier, there is a debate on the validity of SERVQUAL as a service quality measure (Pitt et al., 1995; Kettinger & Lee, 1997; Van Dyke et al., 1997). While SERVQUAL is the most frequently used measure for service quality in IS, it has received some criticism. However, using confirmatory factor analysis, Jiang et al. (2002) found that SERVQUAL is indeed a satisfactory instrument for measuring IS service quality.

Other measures of service quality have included the skill, experience, and capabilities of the support staff (Yoon & Guimaraes, 1995). With the growing popularity of outsourcing for systems development and support, service quality often involves an external provider. The responsiveness of the vendor affects the perception of how ‘cooperative’ that vendor will be (Gefen, 2000).

Measuring use

Empirical studies have adopted multiple measures of IS use, including intention to use, frequency of use, self-reported use, and actual use. These different measures could potentially lead to mixed results between use and other constructs in the D&M model. For example, research has found a significant difference between self-reported use and actual use (Collopy, 1996; Payton & Brennan, 1999). Typically, heavy users tend to underestimate use, while light users tended to overestimate use. This suggests that self-reported usage may be a poor surrogate for actual use of a system. Yet, Venkatesh et al. (2003), for example, found a significant relationship between intention to use and actual usage.

In addition, frequency of use may not be the best way to measure IS use. Doll & Torkzadeh (1998) suggest that more use is not always better and they developed an instrument to measure use based on the effects of use, rather than by frequency or duration. Burton-Jones & Straub (2006) have reconceptualized the systems usage construct by incorporating the structure and function of systems use.

Others have suggested the need to examine use from a multilevel perspective across the individual and organizational levels to enable a better understanding of this construct (Burton-Jones & Gallivan, 2007).

Measuring user satisfaction

The most widely used user satisfaction instruments are the Doll et al. (1994) End-User Computing Support (EUCS) instrument and the Ives et al. (1983) User Information Satisfaction (UIS) instrument. In a comparison between Doll and Torkzadeh's EUCS and Ives' et al. UIS, Seddon & Yip (1992) found the EUCS instrument outperformed the UIS instrument in the context of accounting IS. However, both the EUCS and UIS instruments contain items related to system quality, information quality, and service quality, rather than only measuring overall user satisfaction with the system. Because of this, some researchers have chosen to parse out the various quality dimensions from these instruments and either use a single item to measure overall satisfaction with an information system (Rai et al., 2002) or use a semantic differential scale (Seddon & Yip, 1992). Others have used scales for attitude that are compatible with the concept of user satisfaction (Coombs et al., 2001).

Measuring net benefits

There are an abundance of methods to measure net benefits at both the individual and organizational level of analysis. Perceived usefulness or job impact is the most common measure at the individual level. Yet, there have been occasional problems with the perceived usefulness items (e.g., Adams et al., 1992). Segars & Grover (1993) analyzed the data from the Adams et al. study using confirmatory factor analysis and eliminated an item ‘works more quickly’ in the usefulness construct. In addition, the authors found that ‘job performance’ and ‘effectiveness’ did not fit well with perceived usefulness. The authors used these two items to measure a separate construct called effectiveness. This three-factor construct, perceived ease of use, perceived usefulness, and effectiveness, resulted in a relatively strong fit, as opposed to the poor fit obtained with the original TAM model. Torkzadeh & Doll (1999) have created an instrument to measure different aspects of impact – task productivity, task innovation, customer satisfaction, and management control – to augment their EUCS instrument.

At the organizational level, a variety of measures are employed; but profitability measurements seem to be preferred. The multiple measures for net benefits at each level of analysis make it more difficult to interpret the relationship among some of the success constructs and net benefits. In some studies, the lack of significant findings may be an artifact of measurement, the type of system studied, or some other factor. A key point in terms of measuring organizational benefits, however, is that researchers must ensure that the person evaluating organizational benefits is in a position to answer the questions. Asking users of a system to assess the improved profitability due to the system may not be the best approach. Asking senior managers or referring to objective data from annual reports may be more appropriate when trying to measure organizational benefits.

A comprehensive IS success measurement instrument

Sedera et al. (2004) have developed and validated a multi-dimensional IS success instrument for enterprise systems. This success instrument has been applied and tested in three separate studies. It consists of four dimensions – system quality, information quality, individual impact, and organizational impact – and 27 item measures: nine measures of system quality, six measures of information quality, four measures of individual impact, and eight measures of organizational impact (see Table 2).
Table 2

Validated measures for IS success (Sedera et al., 2004) (used by permission)

System quality

Information quality

Individual impact

Organizational impact

Ease of use

Availability

Learning

Organizational costs

Ease of learning

Usability

Awareness/recall

Staff requirements

User requirements

Understandability

Decision effectiveness

Cost reduction

System features

Relevance

Individual productivity

Overall productivity

System accuracy

Format

 

Improved outcomes/outputs

Flexibility

Conciseness

 

Increased capacity

Sophistication

  

e-Government

Integration

  

Business process change

Customization

   

What makes this particular instrument to measure IS success unique is that this instrument captures the multidimensional and complex nature of IS success by measuring four key success dimensions and by using at least four measures for each dimension. The instrument has strong construct validity in that it captures multiple aspects of each variable, which is a dramatic change from much of the measurement of IS success constructs that focus on only one aspect of the construct. Another strength of this model is that the instrument was rigorously tested within the context of enterprise systems to ensure its validity.

An interesting finding from this research by Sedera et al. is that user satisfaction was eliminated from their success measurement model because it added little explanatory power after the primary four constructs. Use was also eliminated because the system under study was mandatory causing little measurable variation in use.

It is encouraging to see research conducted to create a strong, multidimensional instrument to measure IS success, which overcomes a major shortcoming in previous IS empirical work; namely, inadequate measurement of the dependent variable, IS success. It would be interesting to see if this instrument is relevant to other types of IS beyond enterprise systems. Further research examining to see if use and user satisfaction provide additional explanatory value in different settings, particularly for voluntary systems, would also provide more insight into the measurement of IS success.

Fifteen pairwise comparisons of the IS success constructs

The 15 pairs of relationships shown in Table 1 are discussed in the following subsections. The studies included in this literature review are related to a variety of industries and variety of IS. The only requirements for inclusion in this review of the literature is that the manuscript must (a) report an empirical result (i.e., either quantitative or qualitative) and (b) examine a relationship, broadly defined, within the D&M model. There were no restrictions based on industry, type of information system, mandatory or voluntary systems nature of the system, to better ascertain limitations and boundary conditions for the D&M model.

System quality → use

There is mixed support for this relationship at the individual level of analysis within the literature. Many studies measure system quality as perceived ease of use and find positive relationships with various operationalizations of use in a variety of systems at the individual level of analysis. Perceived ease of use is related to system dependence (Rai et al., 2002; Kositanurit et al., 2006), behavioral intentions to use the system (Venkatesh & Davis, 2000; Venkatesh & Morris, 2000; Hong et al., 2001/2002), extent of use (Hsieh & Wang, 2007) and self-reported use (Adams et al., 1992). Yet other research has found that perceived ease of use is only weakly related to actual use (Straub et al., 1995) and is not significantly related to intention to use (Subramanian, 1994; Agarwal & Prasad, 1997; Lucas & Spitler, 1999; McGill et al., 2003; Klein, 2007), self-reported use (Straub et al., 1995; Gefen & Keil, 1998; Lucas & Spitler, 1999), and system dependence (Goodhue & Thompson, 1995). One study even found, for complex applications such as Lotus 1–2–3, perceived ease of use was negatively related to system use (Adams et al., 1992), suggesting that both system quality and use are complex constructs.

Other research on system quality, using measures besides perceived ease of use, also obtained mixed results. For example, Iivari (2005) found a positive relationship between system quality and use. Goodhue & Thompson (1995) found a significant relationship between reliability and system dependence. Another study identified a significant relationship between perceived ease of use and system usage as measured by the number of different applications used, number of computer-supported business tasks, duration, and frequency of use at the organizational level (Igbaria et al., 1997). Suh et al. (1994) reported a significant correlation between performance of an information system (perceived ease of use, accuracy, etc.) and frequency of use and system dependence. In a study to determine which characteristics of an information system affect intention to use and actual use, Agarwal & Prasad (1997) found mixed results when examining different aspects of system quality, such as relative advantage and compatibility. Venkatesh et al. (2003) found a significant relationship between effort expectancy and intentions to use the system in both voluntary and mandatory settings when measured one month after implementation of a new information system. However, this relationship became non-significant when measured three and six months after the implementation of the system. A case study reporting both qualitative (Markus & Keil, 1994) and quantitative (Gefen & Keil, 1998) results found, however, that perceived system quality did not guarantee the usage of the system. Kositanurit et al. (2006) determined that reliability of an ERP system does not have an effect on utilization of the system by individual users.

Examining the relationship between system quality and use at the organizational level found mixed support for this relationship as well. Caldeira & Ward (2002), in their study of small- and medium-sized Portuguese manufacturing enterprises (SMEs), identified the quality of available software in the market as a factor related to IS adoption and success. One study found that the perceived ease of use of a manufacturing resource planning system did not significantly affect self-reported use (Gefen, 2000). Another study, examining factors related to expert system longevity and use in an organization, noted that technical reasons, such as system quality, were not the main consideration for use or discontinuance of use, which offers further support that system quality may not be a good predictor of use (Gill, 1995). However, a study of the turnaround of the notorious London Ambulance Service Dispatch System failure found that improved system quality was positively related to subsequent system use (Fitzgerald & Russo, 2005). A study looking at IS at a single site using responses from multiple users found that the technical quality of the system was negatively related to use (Weill & Vitale, 1999). Weill and Vitale assumed that this counterintuitive result was probably due to the fact that systems that are heavily used are often fixed quickly, without adequate testing and integration with the current system. This may affect the perception of the technical quality of a system. Premkumar et al. (1994) did not find that the complexity of a system affects the initial use and adoption of an EDI system; however, the technical compatibility of the system with existing hardware and software did affect initial use and adoption of an EDI system.

System quality → user satisfaction

At the individual unit of analysis, there is strong support for the relationship between system quality and user satisfaction (Iivari, 2005). Several types of IS have been examined, and the type of information system affects how some researchers measure system quality. For example, the functionality of a management support information system, which is one measure of system quality, has been found to be significantly related to user satisfaction (Gelderman, 2002). For knowledge management systems, system quality was also found to be strongly related to user satisfaction (Kulkarni et al., 2006; Wu & Wang, 2006; Halawi et al., 2007). For Web sites, system quality, measured as reliability and download time, is significantly related to user satisfaction in two different studies (Kim et al., 2002; Palmer, 2002). Perceived ease of use also has a significant relationship to user satisfaction (Devaraj et al., 2002; Hsieh & Wang, 2007). Researchers have also examined more general IS and found a strong relationship between system quality and user satisfaction using a variety of measures and IS (Seddon & Yip, 1992; Yoon et al., 1995; Guimaraes et al., 1996; Seddon & Kiew, 1996; Bharati, 2002; Rai et al., 2002; McGill et al., 2003; Almutairi & Subramanian, 2005; McGill & Klobas, 2005; Wixom & Todd, 2005). A case study found necessary, but not sufficient, relationships between system quality and user satisfaction and ease of use and user satisfaction (Lexlercq, 2007).

At the organizational level, few studies have examined the relationship between system quality and user satisfaction. Therefore, it is difficult to draw any conclusions on this relationship at this particular level of analysis. One study found that the functionality of executive IS is significantly related to user satisfaction (Benard & Satir, 1993). In two longitudinal case studies, Scheepers et al. (2006) identified a relationship between ease of use of a mobile computing information system and user satisfaction. Premkumar et al. (1994) found no relationship between the complexity of a system and user satisfaction.

System quality → net benefits

The relationship between system quality and net benefits has moderate support within the literature. In general, there is a positive impact on individual performance, although the relationship between perceived ease of use as a measure of system quality and perceived usefulness has seen mixed results. Some studies have a found a significant relationship (Adams et al., 1992; Gefen & Keil, 1998; Agarwal & Prasad, 1999; Lucas & Spitler, 1999; Venkatesh & Davis, 2000; Venkatesh & Morris, 2000; Hong et al., 2001/2002; Devaraj et al., 2002; Yang & Yoo, 2004; Wixom & Todd, 2005; Hsieh & Wang, 2007), while others have found no significant association (Subramanian, 1994; Chau & Hu, 2002; Kulkarni et al., 2006; Wu & Wang, 2006). Seddon & Kiew (1996) and Shih (2004) found that system quality is significantly related to perceived usefulness. Systems reliability and perceived ease of use had no impact on productivity and effectiveness (Goodhue & Thompson, 1995). McGill & Klobas (2005) found no relationship between system quality and individual impact as measured by decision-making quality and productivity.

Kositanurit et al. (2007) identified a significant relationship between perceived ease of use and performance, but no relationship between reliability and performance for individual users of ERP systems. Bharati & Chaudhury (2006) found a significant relationship between system quality, measured by reliability, flexibility, ease of use, and convenience of access, to decision-making satisfaction in an e-commerce environment.

At the organizational level, there exists strong support for the relationship of system quality to net benefits. The quality of an EDI system was found to be related to organizational efficiency, sales, and organizational image (Farhoomand & Drury, 1996). System quality of a data warehouse was associated with decreased time and effort for decision making (Wixom & Watson, 2001). Gefen (2000) also found that perceived ease of use and perceived correctness of software were related to perceived usefulness. The technical performance of an information system was found to indirectly affect the perceived value of the system, mediated by use and user satisfaction (Weill & Vitale, 1999). Another study compared system quality and impact of system use at operational, tactical, and strategic levels (Bradley et al., 2006). The relationship between system quality and impact of use at these various levels was significant. However, when these results were analyzed more closely, it was found that this relationship was not significant at all for formal firms, and only significant at operational levels within entrepreneurial firms.

Information quality → use

Few studies have examined the relationship between information quality and use at both the individual and organizational levels. One reason for this is that information quality tends to be measured as a component of user satisfaction measures, rather than being evaluated as a separate construct. Most of the studies that have examined the relationship between information quality and use focused on IS success models as a whole. Rai et al. (2002) found that information quality is significantly related to use, when use is measured by system dependence. A study of knowledge-management systems found that information (or knowledge) quality was significantly related to intention to use (Halawi et al., 2007). Yet, two studies found that information quality is not significantly related to intention to use (McGill et al., 2003; Iivari, 2005). In their study of task–technology fit, Goodhue & Thompson (1995) found that the quality, locatability, authorization, and timeliness of information were not significantly related to utilization, as measured by system dependence, yet compatibility of information was related to system dependence.

At the organizational level, Fitzgerald & Russo (2005), in their study of the London Ambulance Dispatch System, found a positive relationship between information quality and system use.

Information quality → user satisfaction

The relationship between information quality and user satisfaction is strongly supported in the literature (Iivari, 2005; Wu & Wang, 2006). Studies have found a consistent relationship between information quality and user satisfaction at the individual unit of analysis (Seddon & Yip, 1992; Seddon & Kiew, 1996; Bharati, 2002; Rai et al., 2002; McGill et al., 2003; Almutairi & Subramanian, 2005; Wixom & Todd, 2005; Kulkarni et al., 2006; Chiu et al., 2007; Halawi et al., 2007). Studies specifically examining the information quality aspects of Web sites, such as content and layout, have found significant relationships between these constructs and user satisfaction (Kim et al., 2002; Palmer, 2002). Marble (2003), however, did not find a significant relationship between measures of information quality and user satisfaction of two organizational IS examined in his study.

At the organizational level of analysis, support also exists for the effect of information quality on user satisfaction, but there are not enough studies examining this relationship to reach a strong conclusion. In a qualitative study on system success, data quality and user satisfaction, measured by user attitudes, were found to be directly related to one another (Coombs et al., 2001). Another qualitative case study identified multiple comments from respondents suggesting an association between information quality (i.e., content, accuracy, timeliness, and format) and user satisfaction (Scheepers et al., 2006). A quantitative study also found a significant link between information quality and managerial satisfaction of hardware, software, and support of an information system (Teo & Wong, 1998).

Information quality → net benefits

There is moderate support for the positive impact of information quality on individual performance. Gatian (1994) found that information quality was related to decision-making efficiency. Information quality has also been found to be associated with quality of work and time savings (D'Ambra & Rice, 2001; Shih, 2004) and decision-making satisfaction (Bharati & Chaudhury, 2006). Perceived information quality was also significantly related to perceived usefulness (i.e., a net benefit) (Kraemer et al., 1993; Seddon & Kiew, 1996; Rai et al., 2002; Shih, 2004; Wu & Wang, 2006). Kositanurit et al. (2006) discovered a significant relationship between information quality and performance among users of ERP systems. However, in the context of a knowledge management system, perceived content quality was not directly related to perceived usefulness (Kulkarni et al., 2006). A study of digital libraries found that the relevance of the information retrieved had a significant effect on perceived usefulness, yet the clarity of the terminology used and screen design of the content presented had no relationship with perceived usefulness (Hong et al., 2001/2002).

The relationship between information quality and benefits at the organizational level has shown mixed results, depending on how net benefits are measured. Yet again, more research is needed to reach a conclusion in terms of this relationship. Information quality was found to be significantly related to better perceptions of the work environment (i.e., morale, job content, interesting work) (Teo & Wong, 1998) and to organizational efficiency, sales, and organizational image (Farhoomand & Drury, 1996). Data quality was directly related to perceived decrease in time and effort for decision making in Wixom & Watson's (2001) study. On the other hand, information quality was not found to be significantly related to organizational impact as measured by productivity, competitiveness, and management improvement (Teo & Wong, 1998). Bradley et al. (2006) also studied information quality and the impact of system use in formal and entrepreneurial firms and found largely non-significant results.

Service quality → use

There is little literature that examines the relationship between service quality and use at the individual or organizational level. One study examining this relationship examined accounting IS in Korean firms (Choe, 1996). In this study, the number of years of experience of the IS support personnel was weakly related (P<0.1) to frequency and willingness of use. When analyzed further using Nolan's (1973) Stage Model to measure the maturity of an information system, years of experience of the IS support personnel was significantly correlated with use; however, in later stages of maturity, IS personnel experience was found to be negatively correlated (although not significantly) with usage. This same study examined the role of user training and education and use and found a non-significant relationship between frequency and willingness of use. Again, analyzing the data further using Nolan's Stage Model to determine the maturity of the system, user training and education were significantly related to use in the earlier stages of the information system, but not in the later stages. In another study, documentation of a system was not a predictor of utilization in a survey of ERP users (Kositanurit et al., 2006). A study of knowledge-management systems found that service quality did not predict intention to use (Halawi et al., 2007).

At the organizational level, in the study of the London Ambulance System, the effective role of the technical staff (i.e., service quality) was positively related to the eventual use of the system (Fitzgerald & Russo, 2005). In a study of expert systems, the retention of service staff (and the related funding) to maintain an expert system was a major factor in determining the longevity of the system. Caldeira & Ward (2002), in their study of Portuguese SMEs, found that competency of the support staff, vendor support, and availability of training affected use and adoption of IS.

Service quality → user satisfaction

Several studies have examined the relationship between service quality and user satisfaction; however, the findings of these studies suggest mixed support for this relationship. Researchers have measured service quality using multiple methods, which may account for the inconsistent findings. Some researchers have looked at service quality by examining the characteristics of the support personnel; however, examining the relationship between personnel characteristics and user satisfaction has produced mixed results. Choe (1996) found that IS personnel experience does not significantly affect user satisfaction of accounting IS in Korean firms. Additional analysis of these data, however, noted that if the system was recently implemented, the experience of the IS support personnel was slightly correlated with user satisfaction; yet during later stages of the information system, there was a non-significant relationship between years of experience and user satisfaction of the support team. Another study found that the technical performance of the developers (based on their responsiveness to problems) was positively related to user satisfaction (Leonard-Barton & Sinha, 1993). Yoon et al. (1995) had a similar result in that developer skill had a significant effect on user satisfaction of expert systems. A case study performed by Leclercq (2007) found that the relationship between the IS function and the users as well as the quality of support and services provided by the IS function had an impact on user satisfaction. The mutual understanding between the IS group and the users during the implementation of a project did not have significant impact on satisfaction of the resulting system (Marble, 2003). Chiu et al. (2007) examined the role of support on user satisfaction in an e-learning environment and found a non-significant relationship. Choe (1996) also examined the role of training and education on user satisfaction of an information system and found no significant relationship at any stage of IS implementation.

Examining service quality more broadly, rather than just in terms of personnel and training, there is still mixed support for its effect on user satisfaction. Using the SERVQUAL instrument, which examines the expectations and perceptions that users have on service quality, Kettinger & Lee (1994) found that service quality is positively and significantly related to user satisfaction of information services in a survey of undergraduate students rating the university's computing services department. Another study of university support services found a relationship between service quality and user satisfaction and identified software upgrades, staff response time, and documentation of training materials as the service quality factors having the most influence on user satisfaction (Shaw et al., 2002). Although the two studies in university settings found support for this relationship, a study of 31 government organizations examining internal computing support and user satisfaction did not find a significant relationship (Aladwani, 2002). A study of Web sites found that responsiveness of the site in terms of feedback, assistance, and frequently asked questions was not related to user satisfaction of the Web site (Palmer, 2002). In another Web setting using the SERVQUAL measure, the empathy and assurance aspects of service quality were related to user satisfaction, but not to reliability or responsibility (Devaraj et al., 2002). Halawi et al. (2007) found a significant relationship between service quality, measured using SERVQUAL, and user satisfaction in a knowledge-management context. These findings suggest the sensitivity of the construct of service quality to the manner in which it is measured.

At the organizational level, more research is clearly needed. A qualitative study of system success found that higher quality training and friendly IS support staff led to more positive attitudes about the system (Coombs et al., 2001). Other researchers have examined service quality in terms of vendor support. The role of outside support on user satisfaction has also yielded mixed results. One study found that higher levels of consultant effectiveness and higher levels of vendor support created higher levels of user satisfaction (Thong et al., 1996). In a study attempting to determine the outside support method that yielded higher levels of satisfaction, Thong et al. (1994) found that organizations working solely with a vendor when implementing an IS were more satisfied than organizations using both a vendor and a consultant. Another study examining the use of consultants for selecting and implementing an executive information system found a negative (although not significant) relationship with user satisfaction; the authors suggested that this counterintuitive finding was due to higher expectations that arose when using consultants (Benard & Satir, 1993).

Service quality → net benefits

The relationship between service quality and net benefits has moderate support in the individual context. Igbaria et al. (1997) found that external computing support was related to perceived system usefulness, but that internal computing support was not related to perceived usefulness. Perceived developer responsiveness (Gefen & Keil, 1998) and user training provided by the internal computing department (Igbaria et al., 1997; Agarwal & Prasad, 1999) have been found to be associated with perceived system usefulness. Technical performance of the developers, based on their responsiveness to problems, was positively related to improving efficiency (Leonard-Barton & Sinha, 1993). In a case study on improving service quality, Blanton et al. (1992) found that personalized IT support is more effective than generalized IT support. However, the developer skills for an expert system were not significantly related to the impact on a user's job (Yoon & Guimaraes, 1995). Documentation of ERP systems was also not significantly related to an individual's perceived performance (Kositanurit et al., 2006).

At the organizational unit of analysis more research is clearly needed. Thong et al. (1994, 1996) found that higher levels of vendor support and effectiveness were related to lower operating costs. Gefen (2000) determined that the greater the perception that the vendor is cooperative, the greater the perceived usefulness of a system.

Use → user satisfaction

Surprisingly, little research has examined the relationship between use and user satisfaction. More research has examined the reverse relationship, between user satisfaction and use, so additional research is needed to evaluate this relationship. One study, examining expert systems at DuPont, found that system usage, measured as frequency of use, and user satisfaction, measured using nine items from the Bailey and Pearson instrument, were positively and significantly related (Guimaraes et al., 1996). In a knowledge-management context, Halawi et al. (2007) identified a significant relationship between intention to use and user satisfaction. Seddon & Kiew (1996) found that, in a mandatory context, use, measured by system importance, was not related to user satisfaction. Chiu et al. (2007) identified a significant relationship between use and user satisfaction in an e-learning context. However, Iivari (2005) found, in a study of a medical information system in which use was mandatory, that use measured by amount of daily use and frequency of use was significantly related to user satisfaction. While some researchers have argued that use is irrelevant when a system is mandatory, Iivari (2005) illustrates that it is possible to have sufficient variability in the use construct to have significant relationships with other constructs in the D&M model, such as user satisfaction.

At the organizational level, Gelderman (1998), however, found mixed results in significance in the correlations between different measures of system use (i.e., frequency and duration) and user satisfaction.

Use → net benefits

Empirical studies provide moderate support for the relationship between system use and benefits at the individual level. Several studies have found that IS use is positively associated with improved decision making. Yuthas & Young (1998) found that the duration of system use is correlated with decision performance. Burton-Jones & Straub (2006) found a strongly significant relation between system usage and task performance. Halawi et al. (2007) identified a significant relationship between intention to use and net benefits as measured by improvements in job performance. Many studies confirm these findings by finding significant relationships and/or correlations between system use and net benefits (Goodhue & Thompson, 1995; Yoon & Guimaraes, 1995; Seddon & Kiew, 1996; Abdul-Gader, 1997; Guimaraes & Igbaria, 1997; Igbaria & Tan, 1997; Torkzadeh & Doll, 1999; Weill & Vitale, 1999; D'Ambra & Rice, 2001; Rai et al., 2002; Almutairi & Subramanian, 2005; Kositanurit et al., 2006).

On the other hand, some studies suggest otherwise. One study found that intended use is not significantly related to individual impact (task–technology fit and performance) (McGill et al., 2003). Other studies also found no relationship between use and net benefits (Lucas & Spitler, 1999; Iivari, 2005; Wu & Wang, 2006). Among users in three different Asian firms, there was no significant relationship between frequency of use and job satisfaction (Ang & Soh, 1997). In another series of studies, the self-reported hours of use of IS among managers was positively correlated with decision making in a sample of German firms (Vlahos et al., 2004), but not in Greek firms (Vlahos & Ferratt, 1995).

There is moderate support for the relationship between system use and organizational benefits. Teng & Calhoun's study (1996) found that the intensity of IT usage had a significant impact on job complexity, decision routinization, and decision-making effectiveness. The results of a study of IS use in a hospital setting confirmed a positive relationship between system usage, as measured by the number of DSS reports accessed and number of disk accesses, and profitability and quality of care as measured by decreased mortality (Devaraj & Kohli, 2003). Zhu & Kraemer (2005) found that use of online IS for e-businesses had a positive, significant impact on value in both developed and developing countries. Use of executive IS did impact the productivity, decision-making, and internal costs positively (Belcher & Watson, 1993). Gelderman (1998) also found that system usage, in terms of time duration, was not significantly correlated to revenue and profitability improvement.

User satisfaction → use

Studies examining the relationship between user satisfaction and use have found moderate support for this relationship at the individual level (Iivari, 2005); however, the literature at the organizational level of analysis is lacking. User satisfaction is strongly related to use when measured by system dependence (Rai et al., 2002; Kulkarni et al., 2006), the frequency and duration of use (Guimaraes & Igbaria, 1997; Yuthas & Young, 1998), the number of applications and tasks for which the information system is used (Igbaria & Tan, 1997), and the intention to use (Kim et al., 2002; McGill et al., 2003; Wu & Wang, 2006; Bharati & Chaudhury, 2006; Chiu et al., 2007; Halawi et al., 2007). Also, Wixom & Todd (2005) found a strong relationship between satisfaction and intention to use when mediated by technology acceptance constructs. Winter et al. (1998) found that satisfaction with the system is correlated to both the hours of use and the extensiveness of tasks in a study of white collar workers.

Hsieh & Wang (2007) discovered a significant, positive relationship between satisfaction and extent of use among users of ERP systems in one of their research models that examined the relationships among confirmation of expectations, perceived usefulness, satisfaction, and extent of use; however, this relationship between satisfaction and extent of use became non-significant when placed in a larger model that incorporated perceived ease of use.

Several studies have determined the correlation between user satisfaction and use. However, in these studies we do not know the order of the relationship (i.e., whether use predicts user satisfaction or user satisfaction predicts use). Several studies have found a significant correlation between self-reported system usage and user satisfaction (Abdul-Gader, 1997; Khalil & Elkordy, 1999; Torkzadeh & Doll, 1999). Other studies, however, have found conflicting results. For example, the self-reported hours of use of IS was not significantly correlated to user satisfaction in either German firms (Vlahos et al., 2004) or Greek firms (Vlahos & Ferratt, 1995). Frequency of use was not significantly correlated with user satisfaction in Asian firms (Ang & Soh, 1997). Collopy (1996) found that actual usage of an information system was significantly related to satisfaction; however, self-reported usage was not significantly related to satisfaction.

User satisfaction → net benefits

Empirical results have shown a strong association between user satisfaction and system benefits (Iivari, 2005). User satisfaction has been found to have a positive impact on a user's job (Yoon & Guimaraes, 1995; Guimaraes & Igbaria, 1997; Torkzadeh & Doll, 1999), to improve performance (McGill et al., 2003), to increase productivity and effectiveness (Igbaria & Tan, 1997; Rai et al., 2002; McGill & Klobas, 2005; Halawi et al., 2007), to improve decision making (Vlahos & Ferratt, 1995; Vlahos et al., 2004), and to enhance job satisfaction (Ang & Soh, 1997; Morris et al., 2002). However, Yuthas & Young (1998) found that user satisfaction was only weakly correlated with decision making performance. One study investigated the association between user satisfaction and organizational impact and found that satisfaction was correlated to performance based on profitability and revenues (Gelderman, 1998). Another study found similar results when evaluating the relationship between user satisfaction and organizational performance of ERP systems (Law & Ngai, 2007).

Net benefits → use

At the individual level of analysis, the relationship between net benefits and use has received moderate support. When measuring net benefits, using perceived usefulness as the metric, many studies have found a relationship between behavioral intention and use of a system (Subramanian, 1994; Agarwal & Prasad, 1999; Venkatesh & Morris, 2000; Hong et al., 2001/2002; Chau & Hu, 2002; Malhotra & Galletta, 2005; Wixom & Todd, 2005; Klein, 2007). Other studies have found strong relationships between perceived usefulness and self-reported use (Straub et al., 1995; Igbaria et al., 1997; Yang & Yoo, 2004; Wu & Wang, 2006) extent of use (Hsieh & Wang, 2007) or dependence on an information system (Kulkarni et al., 2006). Venkatesh et al. (2003) found a significant relationship between performance expectancy and intentions to use the system in both voluntary and mandatory settings. These results were consistent over time when measured one, three, and six months after the implementation of the system. In a study by Lucas & Spitler (1999) of brokers and sales assistants at a financial institution, perceived usefulness was not significantly related to intention to use, nor to self-reported use of the system's functionality. Straub et al. (1995) also found no significant relationship between perceived usefulness and actual use. Adams et al. (1992) found both significant and non-significant results in the relationship between perceived usefulness and use, measured by frequency and duration, depending on the information system under study. Another study identified mixed results between benefits and use. Compeau et al. (1999) found positive, significant results when measuring net benefits as performance-related outcomes, which is how the system affects one's job; however, when examining personal-related outcomes, which is intrinsic feelings of accomplishment and the perceptions by others, a small, negative significant effect on computer use, measured by frequency and duration of use, was identified. Agarwal & Prasad (1997) found no relationship between the relative advantage of a system in performing one's work with self-reported frequency of use; however, there was a significant relationship with intention to use.

Examining this relationship from the organizational level of analysis, studies have found strong support for the relationship between net benefits and use. Perceived usefulness is significantly related to self-reported use (Gefen & Keil, 1998; Gefen, 2000). Belcher & Watson (1993) performed an evaluation of executive information systems (EIS) at a single organization using interviews to assess performance and questionnaires to assess system usage. The study found that the benefits of the EIS, increased productivity of workers, improved decision-making ability, and better information flow and connectivity among employees, encouraged continued use of the system. Premkumar et al. (1994) found a significant relationship between the relative advantage of a system measured by the increased profitability, sales, and benefits of a system and the use of a system as measured by initial use of the system and diffusion of use to different types of activities within the firm; however, complexity and compatibility with work processes was not a significant predictor of initial use and diffusion of use of the system. Gill (1996) identified factors such as the type of knowledge required to complete one's work, the variety of tasks, and the identity developed by using the system all had a significant, positive impact on use.

Net benefits → user satisfaction

There is strong support for the relationship between net benefits and user satisfaction. When Seddon & Kiew (1996) tested their version of the IS success model, they opted to replace use with perceived usefulness. Several studies (Seddon & Kiew, 1996; Devaraj et al., 2002; Rai et al., 2002; Kulkarni et al., 2006; Hsieh & Wang, 2007) have found a positive, significant relationship between perceived usefulness (i.e., net benefits) and user satisfaction. A qualitative study also found a relationship between perceived usefulness and user satisfaction (Leclercq, 2007). Three other studies found that the impact an expert system has on a user's job directly affects user satisfaction (Yoon et al., 1995; Guimaraes et al., 1996; Wu & Wang, 2006). These studies, examining the individual unit of analysis, provide support for this relationship. Abdul-Gader (1997) found a significant correlation between perceived productivity and user satisfaction of computer-mediated communication systems in Saudi Arabia. A relationship between decision-making satisfaction and overall user satisfaction was also discovered in a study of e-commerce Web sites (Bharati & Chaudhury, 2006).

At the organizational unit of analysis, however, there is not enough data to comment on the relationship between net benefits and user satisfaction. Two studies examining different aspects of net benefits have yielded similar results. Net benefits, as measured by organizational benefits or impacts, was significantly related to user satisfaction (Teo & Wong, 1998; Jones & Beatty, 2001). Yet, in another study, an IS' effect on the work environment was not significantly related to user satisfaction (Teo & Wong, 1998; Jones & Beatty, 2001). Premkumar et al. (1994) found no relationship between relative advantage (increased profitability, sales, and benefits of a system) and user satisfaction; however, compatibility with an individual's work practices did significantly predict user satisfaction.

Implications and conclusions

Summary of the results

Empirical support for the relationships between the success dimensions are summarized in Tables 3 and 4 and Figures 3 and 4. We classified the level of support for each relationship as strong, moderate, or mixed in order to summarize the empirical results across all studies. To classify the level of support for each relationship, we assigned studies with significant, positive results 1.0 point; studies with both significant, positive results and non-significant results (i.e., mixed results) 0.5 points; and studies with a non-significant relationship between the constructs 0.0 points. The sum of the points was then divided by the number of studies. To assign values of strong, moderate, or mixed, we examined the distribution of percentages. ‘Strong’ support was assigned when the percentage of papers with a positive result was in the range of 90–100%; ‘moderate’ support for a range of 67–83%, and ‘mixed’ support with a range of 25–53%. These percentages are actually quite conservative in that if we used a more stringent quantitative approach, such as vote counting, many of the relationships labeled as moderate and mixed support would show that a significant relationship exists between the constructs (Hedges & Olkin, 1980). However, the purpose of our literature review is not to reduce the relationship between constructs to a number (or effect size), but rather to suggest areas that deserve additional research. To err on the side of caution, any relationship that had four or less studies examining the relationship was deemed as having insufficient data to draw a conclusion.
Table 3

Summary of empirical studies at an individual level of analysis

Relationship

Empirical studies

Study result

Overall result

Conclusion

System quality → use

Halawi et al. (2007)

+

9 of 21 found a positive association

Mixed support

 

Hsieh & Wang (2007)

+

  
 

Iivari ( 2005)

+

  
 

Rai et al. (2002)

+

  
 

Hong et al. (2001/2002)

+

  
 

Venkatesh & Davis (2000)

+

  
 

Venkatesh & Morris (2000)

+

  
 

Igbaria et al. (1997)

+

  
 

Suh et al. (1994)*a

+

  
 

Kositanurit et al. (2006)

Mixed

  
 

Venkatesh et al. (2003)

Mixed

  
 

Agarwal & Prasad (1997)

Mixed

  
 

Goodhue & Thompson (1995)

Mixed

  
 

Adams et al. (1992)

Mixed

  
 

Klein (2007)

NS

  
 

McGill et al. (2003)

NS

  
 

Lucas & & Spitler (1999)

NS

  
 

Gefen & Keil (1998)

NS

  
 

Straub et al. (1995)

NS

  
 

Markus & Keil (1994)

NS

  
 

Subramanian (1994)

NS

  

System quality → user satisfaction

Chiu et al. (2007)

+

21 of 21 positive

Strong support

 

Halawi et al. (2007)

+

  
 

Hsieh & Wang (2007)

+

  
 

Leclercq (2007)

+

  
 

Kulkarni et al. (2006)

+

  
 

Wu & Wang (2006)

+

  
 

Almutairi & Subramanian (2005)

+

  
 

Iivari (2005)

+

  
 

McGill & Klobas (2005)

+

  
 

Wixom & Todd (2005)

+

  
 

McGill et al. (2003)

+

  
 

Bharati (2002)

+

  
 

Devaraj et al. (2002)

+

  
 

Gelderman (2002)

+

  
 

Kim et al. (2002)

+

  
 

Palmer (2002)

+

  
 

Rai et al. (2002)

+

  
 

Guimaraes et al. (1996)*

+

  
 

Seddon & Kiew (1996)

+

  
 

Yoon et al. (1995)

+

  
 

Seddon & Yip (1992)

+

  

System quality → net benefits

Hsieh & Wang (2007)

+

15 of 22 positive

Moderate support

 

Klein (2007)

+

  
 

Bharati & Chaudhury, 2006

+

  
 

Wixom & Todd (2005)

+

  
 

Shih (2004)

+

  
 

Yang & Yoo (2004)

+

  
 

Rai et al. (2002)

+

  
 

Devaraj et al. (2002)

+

  
 

Hong et al. (2001/2002)

+

  
 

Venkatesh & Davis (2000)

+

  
 

Venkatesh & Morris (2000)

+

  
 

Agarwal & Prasad (1999)

+

  
 

Lucas & Spitler (1999)

+

  
 

Gefen & Keil (1998)

+

  
 

Seddon & Kiew (1996)

+

  
 

Kositanurit et al. (2006)

Mixed

  
 

Kulkarni et al. (2006)

NS

  
 

Wu & Wang (2006)

NS

  
 

McGill & Klobas (2005)

NS

  
 

Chau & Hu (2002)

NS

  
 

Goodhue & Thompson (1995)

NS

  
 

Subramanian (1994)

NS

  

Information quality → use

Halawi et al. (2007)

+

3 of 6 positive

Mixed support

 

Kositanurit et al. (2006)

+

  
 

Rai et al. (2002)

+

  
 

Goodhue & Thompson (1995)

Mixed

  
 

McGill et al. (2003)

NS

  
 

Iivari (2005)

NS

  

Information quality → user satisfaction

Chiu et al. (2007)

+

15 of 16 positive

Strong support

 

Halawi et al. (2007)

+

  
 

Leclercq (2007)

+

  
 

Kulkarni et al. (2006)

+

  
 

Wu & Wang (2006)

+

  
 

Almutairi & Subramanian (2005)

+

  
 

Iivari (2005)

+

  
 

Wixom & Todd (2005)

+

  
 

McGill et al. (2003)

+

  
 

Bharati (2002)

+

  
 

Kim et al. (2002)

+

  
 

Palmer (2002)

+

  
 

Rai et al. (2002)

+

  
 

Seddon & Kiew (1996)

+

  
 

Seddon & Yip (1992)

+

  
 

Marble (2003)

NS

  

Information quality → net benefits

Bharati & Chaudhury, 2006

+

9 of 11 positive

Moderate support

 

Kositanurit et al. (2006)

+

  
 

Wu & Wang (2006)

+

  
 

Shih (2004)

+

  
 

Rai et al. (2002)

+

  
 

D'Ambra & Rice (2001)

+

  
 

Seddon & Kiew (1996)

+

  
 

Gatian (1994)

+

  
 

Kraemer et al. (1993)

+

  
 

Hong et al. (2001/2002)

Mixed

  
 

Kulkarni et al. (2006)

NS

  

Service quality → use

Choe (1996)

Mixed

0 of 3 positive

Insufficient data

 

Halawi et al. (2007)

NS

  
 

Kositanurit et al. (2006)

NS

  

Service quality → user satisfaction

Halawi et al. (2007)

+

6 of 12 positive

Mixed support

 

Leclercq (2007)

+

  
 

Shaw et al. (2002)

+

  
 

Yoon et al. (1995)

+

  
 

Kettinger & Lee (1994)

+

  
 

Leonard-Barton & Sinha (1993)

+

  
 

Devaraj et al. (2002)

Mixed

  
 

Chiu et al. (2007)

NS

  
 

Marble (2003)

NS

  
 

Aladwani (2002)

NS

  
 

Palmer (2002)

NS

  
 

Choe (1996)

NS

  

Service quality → net benefits

Agarwal & Prasad (1999)

+

4 of 7 positive

Moderate support

 

Gefen & Keil (1998)

+

  
 

Leonard-Barton & Sinha (1993)

+

  
 

Blanton et al. (1992)

+

  
 

Igbaria et al. (1997)

Mixed

  
 

Kositanurit et al. (2006)

NS

  
 

Yoon & Guimaraes (1995)*

NS

  

Use → user satisfaction

Chiu et al. (2007)

+

4 of 5 positive

Moderate support

 

Halawi et al. (2007)

+

  
 

Iivari (2005)

+

  
 

Guimaraes et al. (1996)

+

  
 

Seddon & Kiew (1996)

NS

  

Use → net benefits

Halawi et al. (2007)

+

16 of 22 positive

Moderate support

 

Burton-Jones & Straub (2006)

+

  
 

Kositanurit et al. (2006)

+

  
 

Almutairi & Subramanian (2005)

+

  
 

Vlahos et al. (2004)*

+

  
 

Rai et al. (2002)

+

  
 

D'Ambra & Rice (2001)

+

  
 

Torkzadeh & Doll (1999)*

+

  
 

Weill & Vitale (1999)

+

  
 

Yuthas & Young (1998)*

+

  
 

Abdul-Gader (1997)*

+

  
 

Guimaraes & Igbaria (1997)

+

  
 

Igbaria & Tan (1997)

+

  
 

Seddon & Kiew (1996)

+

  
 

Goodhue & Thompson (1995)

+

  
 

Yoon & Guimaraes (1995)*

+

  
 

Wu & Wang (2006)

NS

  
 

Iivari (2005)

NS

  
 

McGill et al. (2003)

NS

  
 

Lucas & Spitler (1999)

NS

  
 

Ang & Soh (1997)*

NS

  
 

Vlahos & Ferratt (1995)*

NS

  

User satisfaction → use

Chiu et al. (2007)

+

17 of 21 positive

Moderate support

 

Halawi et al. (2007)

+

  
 

Bharati & Chaudhury, 2006

+

  
 

Kulkarni et al. (2006)

+

  
 

Wu & Wang (2006)

+

  
 

Iivari (2005)

+

  
 

Wixom & Todd (2005)

+

  
 

McGill et al. (2003)

+

  
 

Kim et al. (2002)

+

  
 

Rai et al. (2002)

+

  
 

Torkzadeh & Doll (1999)*

+

  
 

Khalil & Elkordy (1999)*

+

  
 

Winter et al. (1998)*

+

  
 

Yuthas & Young (1998)*

+

  
 

Abdul-Gader (1997)*

+

  
 

Guimaraes & Igbaria (1997)

+

  
 

Igbaria & Tan (1997)

+

  
 

Collopy (1996)

Mixed

  
 

Vlahos et al. (2004)*

NS

  
 

Ang & Soh (1997)*

NS

  
 

Vlahos & Ferratt (1995)*

NS

  

User satisfaction → net benefits

Halawi et al. (2007)

+

14 of 14 positive

Strong support

 

Iivari (2005)

+

  
 

McGill & Klobas (2005)

+

  
 

Vlahos et al. (2004)*

+

  
 

McGill et al. (2003)

+

  
 

Morris et al. (2002)

+

  
 

Rai et al. (2002)

+

  
 

Torkzadeh & Doll (1999)*

+

  
 

Yuthas & Young (1998)*

+

  
 

Ang & Soh (1997)*

+

  
 

Guimaraes & Igbaria (1997)

+

  
 

Igbaria & Tan (1997)

+

  
 

Vlahos & Ferratt (1995)*

+

  
 

Yoon & Guimaraes (1995)*

+

  

Net benefits → use

Hsieh & Wang (2007)

+

15 of 21 positive

Moderate support

 

Klein (2007)

+

  
 

Wu & Wang (2006)

+

  
 

Malhotra & Galletta (2005)

+

  
 

Wixom & Todd (2005)

+

  
 

Yang & Yoo (2004)

+

  
 

Venkatesh et al. (2003)

+

  
 

Chau & Hu (2002)

+

  
 

Rai et al. (2002)

+

  
 

Hong et al. (2001/2002)

+

  
 

Venkatesh & Morris (2000)

+

  
 

Agarwal & Prasad (1999)

+

  
 

Gefen & Keil (1998)

+

  
 

Igbaria et al. (1997)

+

  
 

Subramanian (1994)

+

  
 

Compeau et al. (1999)

Mixed

  
 

Agarwal & Prasad (1997)

Mixed

  
 

Straub et al. (1995)

Mixed

  
 

Adams et al. (1992)

Mixed

  
 

Kulkarni et al. (2006)

NS

  
 

Lucas & Spitler (1999)

NS

  

Net benefits → user satisfaction

Hsieh & Wang (2007)

+

11 of 11 positive

Strong support

 

Leclercq (2007)

+

  
 

Bharati & Chaudhury, 2006

+

  
 

Kulkarni et al. (2006)

+

  
 

Wu & Wang (2006)

+

  
 

Devaraj et al. (2002)

+

  
 

Rai et al. (2002)

+

  
 

Abdul-Gader (1997)

+

  
 

Guimaraes et al. (1996)

+

  
 

Seddon & Kiew (1996)

+

  
 

Yoon et al. (1995)

+

  

aStudies listed in Tables 3 and 4 with an asterisk are studies that have found a correlational association between the constructs, rather than a causal relationship.

Table 4

Summary of empirical studies at an organizational level of analysis

Relationship

Empirical studies

Study result

Overall result

Conclusion

System quality → use

Fitzgerald & Russo (2005)

+

2 of 6 found a positive association

Mixed support

 

Caldeira & Ward (2002)

+

  
 

Weill & Vitale (1999)

  
 

Premkumar et al. (1994)

Mixed

  
 

Gefen (2000)

NS

  
 

Gill (1995)

NS

  

System quality → user satisfaction

Scheepers et al. (2006)

+

2 of 3

Insufficient data

 

Benard & Satir (1993)

+

  
 

Premkumar et al. (1994)

NS

  

System quality → net benefits

Wixom & Watson (2001)

+

4 of 5 positive

Moderate support

 

Gefen (2000)

+

  
 

Weill & Vitale (1999)

+

  
 

Farhoomand & Drury (1996)

+

  
 

Bradley et al. (2006)

Mixed

  

Information quality → use

Fitzgerald & Russo (2005)

+

1 of 1 positive

Insufficient data

Information quality → user satisfaction

Scheepers et al. (2006)

+

3 of 3 positive

Insufficient data

 

Coombs et al. (2001)

+

  
 

Teo & Wong (1998)

+

  

Information quality → net benefits

Wixom & Watson (2001)

+

3 of 4 positive

Insufficient data

 

Teo & Wong (1998)

+

  
 

Farhoomand & Drury (1996)

+

  
 

Bradley et al. (2006)

Mixed

  

Service quality → use

Fitzgerald & Russo (2005)

+

3 of 3 positive

Insufficient data

 

Caldeira & Ward (2002)

+

  
 

Gill (1995)

+

  

Service quality → user satisfaction

Coombs et al. (2001)

+

3 of 4 positive

Insufficient data

 

Thong et al. (1996)

+

  
 

Thong et al. (1994)

+

  
 

Benard & Satir (1993)

NS

  

Service quality → net benefits

Gefen (2000)

+

3 of 3 positive

Insufficient data

 

Thong et al. (1996)

+

  
 

Thong et al. (1994)

+

  

Use → user satisfaction

Gelderman (1998)*

Mixed

0 of 1 positive

Insufficient data

Use → net benefits

Leclercq (2007)

+

5 of 6 positive

Moderate support

 

Zhu & Kraemer (2005)

+

  
 

Devaraj & Kohli (2003)

+

  
 

Teng & Calhoun (1996)

+

  
 

Belcher & Watson (1993)

+

  
 

Gelderman (1998)*

NS

  

User satisfaction → use

No studies

  

Insufficient data

User satisfaction → net benefits

Gelderman (1998)*

+

2 of 2 positive

Insufficient data

 

Law & Ngai (2007)

+

  

Net benefits → use

Gefen (2000)

+

3 of 4 positive

Insufficient data

 

Gill (1996)

+

  
 

Belcher & Watson (1993)

+

  
 

Premkumar et al. (1994)

Mixed

  

Net benefits → user satisfaction

Jones & Beatty (2001)

Mixed

0 of 3 positive

Insufficient data

 

Teo & Wong (1998)

Mixed

  
 

Premkumar et al. (1994)

Mixed

  
https://static-content.springer.com/image/art%3A10.1057%2Fejis.2008.15/MediaObjects/41303_2008_Article_BFejis200815_Fig3_HTML.gif
Figure 3

Support for interrelationships between D&M success constructs at an individual level of analysis.

https://static-content.springer.com/image/art%3A10.1057%2Fejis.2008.15/MediaObjects/41303_2008_Article_BFejis200815_Fig4_HTML.gif
Figure 4

Support for interrelationships between D&M success constructs at an organizational level of analysis.

Table 3 and Figure 3 show the summary of the literature review at an individual level of analysis, while Table 4 and Figure 4 summarizes the results at an organizational level of analysis. The tables and figures clearly indicate that there is a paucity of research examining IS success at the organizational level of analysis. Most of the research relies on individual subjects reporting on their own perceptions about an information system. D&M have claimed that their updated model could be used at any level of analysis. However, more work is needed to confirm whether this is indeed true.

Visual comparison of Figures 3 and 4 shows that, although there are insufficient studies at the organization level to evaluate the strength of most of the relationships, there is consistent support across units of analysis for the following relationships: system quality –> net benefits and net benefits –>use. High-quality systems lead to greater net benefits. Systems that yield higher net benefits are used to a greater degree.

One limitation of any literature review (either a qualitative literature review or a quantitative meta-analysis) is that the findings are highly dependent on the literature identified, examined, and analyzed as part of the review. In an effort to determine whether our results are consistent with other literature reviews, we compared our results to Sabherwal et al.'s (2006) meta-analyses examining IS success. Table 5 compares the results of the literature review (at the individual level of analysis) to the results of Sabherwal et al.'s study.
Table 5

Comparison of literature review results to meta-analyses results

Relationship

Literature review (Individual)

Sabherwal et al. (2006)a

System quality → use

Mixed support

Significant

System quality → user satisfaction

Strong support

Significant

System quality → net benefits

Moderate support

Significant

Information quality → use

Insufficient data

Not examined

Information quality → user Satisfaction

Strong support

Not examined

Information quality → net benefits

Moderate support

Not examined

Service quality → use

Insufficient data

Not examined

Service quality → user satisfaction

Mixed support

Not examined

Service quality → net benefits

Moderate support

Not examined

Use → user satisfaction

Insufficient data

Not examined

Use → net benefits

Moderate support

Significant (correlation)

User satisfaction → use

Moderate support

Not significant

User satisfaction → net benefits

Strong support

Not examined

Net benefits → use

Moderate support

Significant (correlation)

Net benefits → user satisfaction

Strong support

Not significant

aThe results reported here are based on the structural equation modeling assessment that Sabherwal et al. performed on their research model after conducting a series of meta-analyses.

The relationships between system quality and use, user satisfaction, and net benefits all had varying levels of support across both studies. This study also extends the prior work performed by Sabherwal et al. by evaluating the significant relationships among success dimensions that were not examined in their meta-analysis. Relationships associated with information quality and service quality were not examined in Sabherwal et al. but were examined here which supports the proposed relationship between information quality and user satisfaction and between information quality and net benefits. Also, the relationships between net benefits with use and user satisfaction were supported.

The literature on user satisfaction and use has had conflicting results. Our literature review offers moderate support, yet Sabherwal et al. found no support in their structural equation model when examining this relationship. This difference across studies could be due to the different measures that were used across studies for both use and user satisfaction. Probably the most striking finding is that we found strong support for the relationship between net benefits and user satisfaction, while Sabherwal et al. had a non-significant path for this relationship. One difference between these studies is that Sabherwal et al. focused their assessment of net benefits on studies that measured perceived usefulness. Our literature review included studies that used a broader set of measures for net benefits in addition to perceived usefulness. Another value of qualitative research compared to meta-analyses is the ability to understand the direction of association among variables. The empirical studies included in this literature review confirm the direction of the relationships proposed in the D&M IS success model. For example, we find that higher levels of user satisfaction result in greater use but that there is insufficient empirical evidence to know whether more use will lead to greater user satisfaction. Importantly, the literature review also reveals that increased net benefits lead to a higher degree systems use and to a higher level of user satisfaction.

Although a qualitative literature review is subject to the interpretation of its authors, the overall assessments of IS success measurement formed as a result of this review are consistent with, and extend beyond, the results of the quantitative methods that have been employed previously.

Implications for researchers

Measuring success

Although many research studies have tested and validated IS success measurement instruments, most have focused on a single dimension of success such as system quality, benefits, or user satisfaction. Few studies measure and account for the multiple dimensions of success and the interrelationships among these dimensions. Until IS empirical studies consistently apply a validated, multidimensional success measure, the IS field will be plagued with inconsistent results and an inability to generalize its findings. Given that many studies only capture one dimension of each success construct (i.e., perceived ease of use for system quality), measures like those applied in Sedera et al.'s (2004) multidimensional success measurement instrument provide higher content validity. Their research has proven to be a valid and reliable step toward improved IS success measurement and either their instrument or their approach for creating and validating instruments should be adopted and further tested in different contexts.

Another problem for IS success measurement is the proclivity to use only the user satisfaction dimension as a surrogate measure of success. Studies have found that self-reported measures are not consistent with actual measures (Heo & Han, 2003). Subjective measures are therefore not always a very reliable substitute for objective measures of success. Nevertheless, many empirical studies of IS effectiveness adopt user satisfaction as a primary measure or surrogate of IS success due partially to convenience and ease of measurement.

One finding in this research is that the various measures of success used in the studies examined for the literature review could be classified using one or more of the dimensions of the D&M IS success model. This is clearly a strength of the D&M model; however, several studies could not be classified in the literature review because of the use of a general effectiveness measure that measures multiple dimensions of success (Sedera et al., 2004). The utilization of user satisfaction as a surrogate for success masks the important dimensions of success such as system quality, information quality, and net benefits. Generally, most researchers do not take the time to parse the different dimensions from the user satisfaction instrument to measure specific dimensions of success. User satisfaction in and of itself is often a goal for IS and therefore worthy of measurement, but satisfaction should not be used alone as the sole indicator of success. This is part of the reason for the lack of studies examining the relationship between information quality and other success constructs.

The results of the literature review reported here emphasize the current problems the field is experiencing with measuring and understanding system use. The relationship between each dimension and use is lower than associations with any other dimension; that is, system quality and use has less support than system quality and user satisfaction and system quality and net benefits. The inconsistency regarding the use construct seems to be largely related to the measurement and understanding of the use construct. System use is a success construct that is often criticized and/or ignored. Its significance has found mixed results in empirical studies. It is our contention that system use is an important indicator of IS success and associated with the ultimate impact or benefits garnered from IS. Much like the measurement of success as a whole, we believe that the measure of system use has been over simplified; ignored when use is mandatory and poorly measured as merely frequency or time of use when voluntary.

Use is not a dichotomous variable on the volition scale; that is, it is rarely ever either totally voluntary or totally mandatory. A three-year case study of eight organizations found that IS use within the organization was not simply a question of use or non-use, but rather that the utilization was complex, with transition and equilibrium periods (Lassila & Brancheau, 1999). Also, perceived use is not a very satisfactory method for measuring use. Intensity, purpose, sophistication, extents, and more should be included in the study of use. Researchers should always include use as a measure of success and omit it only after empirical results demonstrate that it provides little or no added explanatory value beyond the other dimensions of success in the study under consideration.

A final issue associated with the measurement of constructs is the dependence on measures from TAM. First, content validity suffers because system quality is only measured in terms of ease of use and net benefits is only measured in terms of perceived usefulness. In the studies examining the relationship between system quality and use, there was mixed support for this relationship if the researcher used only perceived ease of use as only the measure of system quality. Measuring system quality using measures other than, or in addition to, perceived ease of use more consistently predicted the different measures of use.

More importantly, this focus on TAM keeps researchers focused at the individual level of analysis. Since most studies have focused on the benefits to an individual, we have little understanding of the impact of an information system at an organizational level as found and described in this study. Researchers need to make a conscious effort to consider studying the concept of net benefits beyond the individual and consider group, organizational, and even possibly societal impacts.

Understanding the relationships

By separating the studies examining the individual context from those examining the organizational context, some interesting findings come to light. First, it was discovered that there are few studies that have considered the relationships that comprise the D&M model from an organizational point of view. Only three of the 15 relationships among the success dimensions have received any reasonable level of study. It was also clear that by grouping studies in terms of their context (i.e., individual or organizational), it was possible to better understand some of the relationships. For example, if the studies are aggregated across contexts for system quality and benefits, there is moderate support. However, this grouping of studies reveals that there is stronger support for this relationship at the organizational level than at the individual level. This same phenomenon occurs for the relationship between benefits and use. The organizational studies of this relationship have more consistently found a relationship between these constructs than studies at the individual level. If, for example, the studies for information quality and use are aggregated, the result is moderate support; however, careful parsing of these studies by context shows that there are really insufficient data to evaluate these relationships at the individual and organizational levels.

This qualitative literature review did not restrict studies by the mandatory or voluntary nature of the system use, or by the type of information system examined. It may be that relationships with strong support are not subject to boundary conditions such as the voluntariness of the system or the type of information system; however, more research is needed to confirm this conjecture.

Studies have shown that service quality is associated with individual performance; therefore, service quality deserves to be included as a dimension of IS success. Despite some controversy, SERVQUAL has been shown to be a valid measure of IS service quality. Failure to consider system quality, information quality, and/or service quality can lead to confounding results where the implied negative impact of an independent variable may actually be the result of poor systems-related dimensions that have not been taken into account.

Finally, information quality is a neglected aspect of IS success. Since a primary motivation for IS applications is to supply managers with accurate, timely, and relevant information, information quality is an important dimension of a system's success. Information quality is also an important factor in measuring user satisfaction and should be treated as a success measure separate from the popular end-user satisfaction instrument (Doll et al., 1994). Since information relevance is an important dimension of information quality and can vary widely by systems, there is likely to be high variance in information quality in practice; therefore, this variance should be accounted for in empirical IS research (Sedera et al., 2004).

Implications for practice

Practitioners consistently acknowledge the importance of measuring the value of their IS investments. However, practitioner IS-effectiveness measurement methods suffer the same deficiency as academic success models; that is, they are often one-dimensional and over-simplified. Practitioners tend to focus on net impacts or benefits but fail to consider system, information, and service quality as well as the nature and intensity of system use.

As for the important net benefits dimension, Kaplan and Norton's ‘Balanced Scorecard’ (1996) holds promise for measuring the business contribution of IS. Martinsons et al. (1999) proposed a balanced IS scorecard that consists of four performance dimensions: namely, a business-value dimension, a user-oriented dimension, an internal-process dimension, and a future-readiness dimension.

According to a study by Heo & Han (2003), the constructs of the IS success model have different degrees of importance depending on the firm characteristics. Firms with more centralized computing place emphasis on performance measures in the following order: system quality, information quality, user satisfaction, use, organizational impact, and individual impact. Firms with decentralized computing environments emphasize information quality and system quality highly; emphasize user satisfaction, individual impact, and organizational impact moderately; and information use as the least important measure. For firms with centralized cooperative computing, organizational impact and system quality were the most important measures, followed by information quality, user satisfaction, individual impact, and use. For firms with distributed cooperative computing, organizational impact is the most important factor as well as individual impact and information quality. Use was found to be the least appropriate measure for this group.

In another study of the importance of various IS success factors to managers, Li (1997) found that accuracy of output, reliability of output, relationship between users and the CBIS staff, user's confidence in the systems, and timeliness of output were the most important factors. The five least important factors were the chargeback method, volume of output, competition between CBIS and non-CBIS units, features of the computer language used, and job effects of computer-based support.

Rainer & Watson (1995) studied factors associated with the success of executive IS, using the D&M model, and found ease of use, information quality, and impact on work were critical factors in these types of systems.

Practitioners are advised to deploy success measurement programs that incorporate all six dimensions of IS success: system quality, information quality, service quality, objective (as opposed to subjective) measures of system use, user satisfaction, and net benefits. The context, purpose, unit of analysis (individual vs organizational), and importance of systems should dictate the relative weights to place on each of these success dimensions and measures. An IS balanced scorecard should also be considered as a way of measuring net benefits.

Future research

While recent research has provided strong support for many of the proposed interrelationships among success dimensions in the D&M model, more research is needed to explore the relationships that have not been adequately researched. Empirical research is also needed to establish the strength of interrelationships across different contextual boundaries. This study takes a first step by parsing out the results based on individual vs organizational units of analysis and found that there is insufficient empirical evidence to evaluate most of the relationships at the organizational level. However, there could be other, more complex effects that could explain the relationship between these success constructs at either the individual or organizational levels of analysis. Researchers may want to consider complex functions, such as curvilinear effects, that affect the relationships among IS success constructs.1 There are also a number of other boundary conditions that deserve attention, such as the voluntariness of the system, the timing of success measurement (i.e., the difference between the time of the implementation of the system and the time of measurement), and the type of information system examined. Secondly, more research is needed on the relationships between information quality and use, user satisfaction, and net benefits. Finally, IS researchers still struggle with system use as a measure of IS success. Future studies must apply more comprehensive and consistent measures of use in order to better understand the effect of use on user satisfaction and net benefits. Burton-Jones & Straub (2006) have taken an important step in improving the measurement of systems use by incorporating the structure and function of use.

Conclusion

This literature review examined IS success at both individual and organizational levels of analysis. The D&M IS success model applied equally well at both the individual and organizational levels of analysis in those cases where there were sufficient data to analyze the relationships. This research also considered many different types of IS under a variety of conditions and had reasonable support for the majority of relationships within the model, suggesting the value of the D&M model of IS success when evaluating utilitarian IS. What still remains to be discovered is if the D&M model is appropriate for hedonic IS. Some of the dimensions may no longer be relevant or may need to be measured differently for gaming, social networking, or other types of IS used for enjoyment.

The science of measuring information success or performance in empirical studies has seen little improvement over the past decade. Researchers and practitioners still tend to focus on single dimensions of IS success and therefore do not get a clear picture of the impacts of their systems and methods. Progress in measuring the individual success dimensions has also been slow. The work of Sedera et al. (2004) in developing measures for success is encouraging and this type of work should be continued in future research. Valid and reliable measures have yet to be developed and consistently applied for system quality, information quality, use, and net benefits. The D&M IS success model (2003) is a useful framework for understanding the key success dimensions and their interrelationships. However, researchers must take a step further and apply rigorous success measurement methods to create comprehensive, replicable, and informative measures of IS success.

Footnotes
1

We thank an anonymous reviewer for suggesting this point.

 

Copyright information

© Palgrave Macmillan Ltd 2008

Authors and Affiliations

  1. 1.Department of Information Systems and Quantitative Analysis, University of Nebraska at OmahaOmahaU.S.A.
  2. 2.Department of Information Technology, American UniversityWashingtonU.S.A.
  3. 3.Department of Computer Information Systems, Robinson College of Business, Georgia State UniversityAtlantaU.S.A.