Approach to Evaluating the Effect of an Inter-organizational Information System on Performance: The Case of a Destination Management Organization

Abstract

This research proposes an approach to evaluate the contribution of an interorganizational information system (IOIS) to processes and organizational performance. Using a process-based framework, the approach was developed from a review of the IS evaluation literature and then refined through an in-depth embedded case study of an IOIS used by a destination management organization (DMO). The need for this research, comes from the significant investments in terms of capital and human resources and the numerous challenges that IOISs represent for DMOs. DMO’ IOISs are characterized by their interdependence between multiple stakeholders with sometimes contradictory interests. The approach developed here is of interest to researchers and practitioners in that it allows for a contextualization of IOIS system evaluation, and that it considers the depth and breadth of performance measures.

Keywords

1 Introduction

As a catalyst for the economic development of the destination, the DMO assumes a strategic leadership role [1] with its stakeholders, tourism service providers and business authorities in order to develop content and programs related to a strong tourism image [2]. To carry out this mission, the DMO must acquire various technologies. As early as 1997, Chen and Sheldon [3] indicated that it is often in a complex relational context and subject to contradictory forces that DMOs are led to acquire an interorganizational information system (IOIS). IOISs support information exchange and business transactions between different organizations and are based on networks that transcend organizational limits [4, 5]. They focus on the optimization of the internal processes and on relations within the destination’s stakeholders. There is a lack of consensus on the conceptualization of IOIS used by DMOs, and they are often studied as DMO websites [6]. Previous research focused on antecedents of DMO systems adoption, on implementation drivers and on expected benefits. DMOs continue to invest in the development of their IS [7,8,9] while the benefits are neither guaranteed nor easily understood [10,11,12]. Despite the numerous studies on the IOISs used by DMOs, questions remain about the disappointing performance of these systems [13]. The majority of research does not address the challenges posed by transactional and inter-organizational components [5, 14]. However, in the era of smart destinations, DMOs need to have a global vision supported by interoperable and interconnected systems [11, 15, 16]. Hence, studying how to evaluate the performance of IOISs used by DMOs, while considering the research needs found in the literature, seems relevant. Sigala [13] proposes an evaluation that takes into account the inter-organizational context and the vision of multiple stakeholders. Buhalis, Leung [17] highlight the strategic usefulness of the IOISs used by DMOs to support the differentiation efforts of tourism SMEs and to strengthen the competitiveness of the destination.

This research aims to propose a qualitative method for evaluating the performance of an IOIS. Using a process-based approach, this method, issued from an IS evaluation literature review, is applied to a DMO’s IOIS as an embedded case study.

2 Literature Review and Theoretical Framework

Given the diversity of its theoretical foundations, the research on the ex-post evaluation of information systems presents several models and still reports difficulties in assessing the impact of IOISs [18]. The evaluation of ISs poses challenges, both conceptual and methodological. Conceptual challenges can be classified into four categories: (1) biases related to the chosen theoretical framework [19, 20]; (2) heterogeneous conceptualization of the evaluation variables [21, 22]; (3) the intangible and hidden nature of costs and benefits [23] and, finally, (4) the anchoring biases related to the benchmark chosen to evaluate the IS. Also, the complex nature of organizational performance is often conceptualized in only productivity or profitability dimensions whereas it is multi-dimensional and dynamic [24].

Research on the effects of ISs relies primarily on two types of models: causal models and process models. Causal models attempt to demonstrate a linear and unidirectional and static relationship between IS investments and organizational benefits [18, 19]. The parsimony of these models explains their frequent use in IS evaluation. However, they produce conflicting results regarding the IS value paradox [20]. Process-based models, on the other hand, measure the effects of IS at the level of intermediate processes before evaluating their impact on organizational performance [25, 26]. They explore the chain of events by which the IS investment is converted into added value to the organization. Despite a greater level of complexity and more difficult operationalization, process-based models are known for their ability to render empirical reality [18]. This is the reason for their choice as the basis of the conceptual framework of this research in combination with the performance prism developed by Neely, Adams [24], to evaluate the effects of a DMO’s IOIS on the DMO performance.

The theoretical framework, as shown in Fig. 1, states that the performance of DMOs’ IOISs is based on several concomitant and competing factors: the strategic objectives pursued by the DMO, the capabilities implemented by all stakeholders, their use of the IOIS, the effects of the IOIS on processes, and the nature of their contribution to IOIS performance. The theoretical framework, based on the resource-based view theory of the firm, combines process-based models for evaluating IS performance [25, 26] with the integrated organizational performance evaluation method of Neely, Adams [24]. As such, it responds to the recommendations of Marthandan and Tang [27] regarding the need to use multidimensional methods combined with multi-perspective approaches for the efficient evaluation of IOISs. The prism of the performance of Neely, Adams [24] was designed to take into account a significant number of stakeholders.

Fig. 1.

Theoretical framework for evaluating the performance of an IOIS

The theoretical framework consists of six blocks addressing: (1) the characteristics of the IOIS and the technological capabilities of the DMO and its complementary capabilities as defined by Melville, Kraemer [25]; (2) the capabilities and expectations of the stakeholders external to the DMO following the logic of Melville, Kraemer [25] and Neely, Adams [24]; (3) the operational and management processes supported by the IOIS; (4) the automation, information and transformational effects of the IOIS on the processes [26]; (5) performance measures of operational and management processes [24]; and (6) satisfaction and contribution of stakeholders using IOIS [24]. Stakeholder contribution defines the behavior that the DMO expects its stakeholders to have in relation to the IOIS. Satisfaction is an individual measure of process performance. According to Sabherwal, Jeyaraj [28], organizational variables (voluntary use of IOIS by tourism organizations) and individual variables (mandatory use of IOIS by DMO employees) may coexist within the same framework.

The theoretical framework for evaluating the performance of an IOIS served as a guide in the empirical part of the research, whose main question was: How can the performance of a destination’s IOIS be evaluated? This main question considered the performance of the IOIS according to each stakeholder and suggested the specific question #1 “How do the dimensions of the DMO’s IOIS performance differ according to the stakeholder concerned?” Finally, the main question also focused on the integration of these dimensions into a whole, which led to the specific question #2 “How can these dimensions be integrated into a tool for evaluating the performance of a DMO’s IOIS?”.

3 Methodology

In order to address the methodological difficulties of IOIS evaluations, it is important to situate oneself among the various existing types of evaluation. This research is a participating ex post evaluation of IS organizational impact. As an inductive evaluation, it avoids evaluations’ anchoring bias of preset objectives or criteria. With the performance prism of Neely, Adams [24], an integrated evaluation method was chosen rather than quantitative financial or economic ones.

Users of the IOIS participated in the evaluation and the researcher adopted a consultant’s position conducting a summative evaluation [24]. The evaluation was conducted from the perspective of the DMO, taking into account internal stakeholders (DMO employees) and external stakeholders (tourism organizations that are members of the destination) [13]. Finally, the level of analysis used is that of the impact of the IOIS on the following processes.

For the interpretivist, the process of knowledge creation involves understanding the meaning that actors give to reality. The validity of this knowledge depends on the idiographic nature of the data and the empathic perspective of the researcher. The research design was therefore developed in such a way as to produce rich, contextualized data for discovery purposes. This research uses an exploratory qualitative approach based on inductive reasoning applicable to the discovery of new phenomena. The case study method was chosen to answer a descriptive research question. A single case with multiple groups (the embedded cases) was chosen to conduct this research. This method facilitates a new understanding [29] of the performance of an IOIS from a perspective that has not yet been addressed in the literature. The single case study has sub-divisions where attention is focused in particular ways, understood as the different perspectives of stakeholder groups. Tourisme Montréal’s IOIS was used as the case for a contextualized exploration of the theoretical framework (Fig. 1). This type of sample provides instrumental interest and meets the criteria of theoretical representativeness of the case and richness of the available data [29]. Tourisme Montréal is a regional DMO, which, in North America, is the level with the largest number of DMOs that own and manage their own ISs. Tourisme Montréal is a private non-profit organization that brings together members from various tourism sectors: accommodation, attractions, tourism facilities, events and various services. Tourisme Montréal’s mission consists of “maximizing the economic benefits of tourism, uniting the local tourism industry, deploying innovative tourist welcome strategies, as well as developing the tourism offer.” As such, it uses its IOISs to disseminate information on the destination’s offer and to enable the online reservation of products and services. It also uses IOISs to build a customer database and deploys it in its sales and marketing department. The system also makes it possible to offer monitoring and training services to participating organizations.

Stakeholder Groups:

At this level, theoretical representativeness implies a homogeneity of respondents in terms of the issue to be studied. In this case, the stakeholder groups concerned all had to be users of Tourisme Montréal’s IOIS. These were, internally, employees and managers, and externally, tourism organizations that are members of Tourisme Montréal and active in the accommodation, museum or travel agency sectors. The purpose of this research was not to gather the consumer’s point of view directly, there already being a great deal of literature on this subject. Consumer satisfaction is intrinsic to the definition of performance [24] and, as such, it is the main objective pursued by each group of stakeholders.

The Respondents:

The number and type of companies to be surveyed within each group of external stakeholders, respect the criteria of variety and heterogeneity. The sample of internal respondents is non-probabilistic, voluntary and purposive. The concept of informational power, linked to the specific objectives of this research, also guided the composition of the respondent sample [30]. The eighteen internal respondents and the eight external ones were targeted based on their ability to comment on the performance dimensions of this IOIS. External respondents were Tourisme Montréal members from hotels, travel agencies and attractions sectors. Internal respondents were Tourisme Montréal employees from sales, convention servicing, communications, marketing, members service, research and product development departments. The multiplicity of stakeholder groups as well as the variety of hierarchical positions within the DMO between coordinators, managers, directors and vice-presidents helped to triangulate information sources and enrich data collection.

Data Analysis:

The twenty-six pieces of verbatim data were coded using Atlas.ti software. A first coding allowed the researcher to become familiar with the data from the first interviews. It focused on the meaning that emerged from the words used by the respondents [31]. In a first step, the codes were used to determine the code tree structure. Subsequently, thematic coding [32] was performed with predefined codes based on the literature and corresponding to the categories included in the evaluation model. It organized the data according to seven categories which were divided in the following sub-categories: capacities, expectations (system quality, information quality, service quality, communication needs, training needs, networking needs), processes (sub-processes and task), IOIS effects (automational, informational, transformational), processes performance indicators, satisfaction and contributions. Finally, a second coding was applied to the content of each major category, using in vivo codes. Partially ordered matrices were used to reveal indicators’ similarities and differences per stakeholder groups according to the processes evaluated. For each process, matrices included the expectations, the satisfaction, the IOIS effects and the processes performance indicators. A logbook and the memos made it possible to track the iterations of the analysis. Theoretical saturation, in terms of meaning, was achieved for each stakeholder group [33].

4 Results and Discussion

This research has made it possible to demonstrate the usefulness of the theoretical framework developed for evaluating the performance of an IOIS and, to propose an approach for evaluating the performance of an IOIS. Figure 2 below illustrates how the dimensions of the performance of the DMO’s IOIS differ according to the internal and external stakeholders involved (answer to specific research question #1).

4.1 Dimensions of IOIS Performance for Internal Stakeholders

The top of Fig. 2 shows the dimensions and performance indicators of the DMO’s IOIS that relate to its internal stakeholders: (1) the capabilities of the DMO and its employees, (2) the expectations of DMO employees with respect to the processes supported, or not, by its IOIS, (3) the automation, information and transformation effects of the IOIS on DMO processes, (4) the satisfaction of DMO employees with the IOIS, and (5) the contribution of employees to the DMO’s IOIS.

Capacities of the DMO and Its Employees.

This first dimension reflects the necessary alignment of the DMO’s strategies with the processes supported by the IOIS, its capabilities as a DMO and the individual capacities of its employees. This alignment provides an understanding of how the DMO can articulate its strategies and the processes supported by its IOIS toward the satisfaction of its employees and in accordance with their expectations and those of the DMO. DMO’s capacities include (1) the capacities of the DMO’s technology department, (2) the individual technological capacities of the employees, (3) the complementary organizational resources (human resources, material and financial resources, training provided) of DMO, (4) the strategic directions of the DMO, and (5) the characteristics of the IOIS and its functionalities. Although the use of the IOIS is mandatory, this does not mean that it is optimal in all departments and for all automated processes.

For example, a difference in the training provided on IOIS functionalities was at the origin of significant variations between departments.

Internal Stakeholder Expectations.

The tool provides measures of internal stakeholder expectations that relate to the time required to complete a process and the costs involved. Internal stakeholders are concerned about improving the quality of the system: “Now all departments are ready for a new IS.” This includes ease of use and access, speed of response and execution, intuitive navigation “like Google,” flexibility of queries, visual appearance and formatting to suit needs. At the same time, they seek to improve the quality of information, based on the criteria real time, up-to-dateness, reliability and completeness: “to become a nerve center of information.” Some internal stakeholders are in favor of integrating functionalities that allow the IOIS to be adapted to relational marketing, to promote the integration of technologies with one another: “a single platform for the entire destination.” Participants also mentioned the need to make better use of all the functionalities of the IOIS and the need to improve the quality of the IS service to support users.

Processes Supported by the IOIS.

In the process of evaluating an IOIS, it is necessary to understand its impact on process performance. To this end, this dimension lists all the processes supported by the IOIS. Throughout the interviews, details are provided regarding the level of automation of all the tasks that make up these management and operational processes.

Effects of the IOIS, Satisfaction, Contribution and Performance Indicators.

Automational, informational and transformational effects are considered when evaluating the IOIS by the internal stakeholders. First, the automational effects of the IOIS are related to the quality of the system, its navigation, stability, processing speed, response time, “just a click away” and remote accessibility. The DMO’s employees mentioned automational effects such as loss of autonomy, work overload, “more steps than before” and the complexity of tasks to be performed in the system. The fact that the expectations of a certain group of users had not been considered during implementation explains these adverse effects. It should be also noted that some sub-processes may be supported differently by the IOIS and cause opposite impacts.

Fig. 2.

Evaluation of Tourisme Montréal’s IOIS performance

Second, the informational effects of the IOIS relate to the quality of the information and the system’s accuracy, updating, presentation, level of centralization and shareability, and relevance. With regard to the respondents’ responses, the measurement of the time spent carrying out a process and the costs this entails must be included in the performance indicators as they influence user satisfaction. Risk management related to information ownership is reduced when vendors are able to update information remotely in the IOIS. For the DMO, such quasi-simultaneous access to this information means that it can attain ownership of the information and, submit an estimate to potential customers at great speed. Thirdly, transformational effects occur immediately after a new IOIS implementation or new functionality is implemented, since a re-engineering process is underway. This is the case when certain processes are outsourced to DMO members, such as the updating of tourism establishment information and the submission process of group quotes. Some DMO departments can become more autonomous in carrying out certain processes, as was the case with the sending out of the newsletter to Tourisme Montréal members. They no longer had to wait for the communications department to extract information from the database, as a result of which members’ newsletters were issued more frequently, and with up to date information: “awesome, more interesting, motivating to develop these tools; real time marketing.” Transformational effects therefore also have an impact on the perceived value to external stakeholders. Finally, process re-engineering can change the level of autonomy of managers versus coordinators, due to their different abilities to work on a new system: “It’s reassuring to know that it’s available to everyone and that it isn’t just me who knows how to fly the airplane.” However, these transformational effects are expected to be temporary as “data volume grows.”

Contributions Expected from Employees.

This dimension of the tool measures the extent to which DMO employees are involved in using the full potential of the IOIS, such as the constant suggesting of improvements, their regular training on the database, and their possible actions as ambassadors of the online destination: “super users.”

Performance Measures of the Processes Supported by IOIS.

The analysis shows that the indicators used by DMOs measure performance at a global level and are not always related to the effects of the IOIS on processes. For example, at Tourisme Montréal, the client satisfaction rate was measured for the entire destination and does not reflect on the performance of services offered through the DMO’s IOIS. Nevertheless, the sales department had some relevant performance indicators at the process level, such as: the number of leads generated, the number of follow-ups and development actions carried out, the number of coordinated tracking visits, and the number of newsletters sent. It has proven useful to be able to calibrate the costs related to the execution of certain processes according to the time spent on an action. This would be a good indicator of the need to automate or transform certain processes supported, or not, by the IOIS. However, the process/time/cost indicators are missing in this case study.

Furthermore, in this case study, the impact of the IOIS on the overall performance of the DMO could not be assessed due to the absence of performance measurement indicators that link process efficiency (process-level performance indicators) to the overall performance indicators of the DMO. This finding confirms that the effects of technologies on processes do not necessarily correspond to the performance indicators used by process managers [18].

4.2 Dimensions of IOIS Performance for External Stakeholders

At the bottom of Fig. 2 are dimensions of IOIS performance that relate to external stakeholders, in other words, DMO member organizations.

Capacities of DMO Member Organizations and Their Use of IOIS Functionalities.

As with the capacities of the DMO and its employees, this first dimension reflects the necessary alignment of the DMO’s strategies with the processes supported by its IOIS and the capacities of DMO member organizations. This alignment provides an understanding of how the DMO can gear its strategies and processes supported by its IOIS toward the satisfaction of external stakeholders, in accordance with its own expectations. The capacities of member organizations include their own resources as well as those of their brand’s head offices. It is the availability of these resources that conditions to what extent these organizations really need the DMO’s IOIS.

Effects of IOIS on the Processes, Satisfaction and Expectations of External Stakeholders.

There are, to begin, the IOIS automational effects, which can be measured for external stakeholders in terms of the quality of the system, ease of navigation, ergonomics, speed and interactivity of the system, and flexible formatting of the reports to suit needs. External user satisfaction is positive if the value of the previous measurement indicators exceeds their expectations. It will be negative if not.

The expectations of the external stakeholders can guide the DMO IOIS deployment strategies. Some external stakeholders use IOIS functionalities out of solidarity with the DMO rather than to respond to a real need. When evaluating the IOIS at the process performance level, and hence not only at the level of solidarity use, it became obvious that the individual hotel reservation process was cumbersome, complex and caused errors and loss of time for its users:: “I’m all behind Tourisme Montréal; but it’s not in any way a booking engine, it’s for promotion.” External stakeholders do not all attach the same importance to process automation seeing that they do not all use them in the same way. Expectations for automation are high among hoteliers.

Secondly, the indicators for measuring the informational effects of the IOIS for external stakeholders focus on the quality of the information, in other words, its timeliness, completeness and relevance of its content, timing and conciseness. The sometimes-contradictory interests of these external stakeholders appeared in terms of informational effects because of their different information needs: “long stays markets,” “deluxe leisure markets,” “increased paid visibility” or an “excess of information; we don’t have time to stroll around in members’ extranets.”

Finally, transformational effects are rarely encountered by external stakeholders, as the study revealed, except for the individual reservations process that demanded considerable adjustments on behalf of the hotels.

External Stakeholder Contribution.

For each expectation of the external stakeholders toward the IOIS, a counterpart is expected by the DMO, in the form of a contribution toward the IOIS and/or a voluntary participation. This is a key dimension of the performance prism. All uses of the IOIS by the external stakeholders are voluntary. One member qualifies his relationship with the DMO as “intuitive” and has the impression that he isn’t making “enough use of it due to a lack of time.” Indicators to measure the contribution of external stakeholders to the DMO can therefore be, the frequency of use of the IOIS, the overall value of these uses (member fees, commissions on individual and group bookings, promotional placements, entrance to networking activities, etc.); the average value per transaction; the frequency of suggestions for improvement of the IOIS; and the perceived value of the services offered by the DMO’s IOIS.

4.3 An Approach to Evaluate the Performance of a DMO’s IOIS

The process-based evaluation approach used in this case study brings together all the needs and expectations of external and internal stakeholders by linking them to processes supported by the IOIS (answer to specific research question #2). The process-based approach was found to be the most likely to reflect the mechanisms by which the IOIS could affect the performance of the DMO: the exploitation of the IOIS in the operational processes of the DMO produces automational, informational and transformational effects which, in turn, affect the performance indicators at the level of the operational processes used by the stakeholders. These processes performance indicators have an influence on the overall DMO performance.

If the evaluation process reveals that the IOIS has little or no impact on the overall performance indicators used by the DMO managers in managing their processes, the following questions should be asked:

• Is the IOIS used appropriate for the DMO? Do its functionalities meet the expectations of the stakeholders? Are the key processes supported by the system?

• Do the DMO’s IS strategies meet the needs of the stakeholders?

• Is the implemented IOIS being exploited to its full potential?

• Have investments been made in the “complementary assets” (DMO’s capabilities) that enable the development of the IOIS? Does the capacity of user tourism organizations allow the IOIS to support the processes?

• Do the performance indicators used by the DMO truly reflect its organizational performance in all its aspects and capture the efficiency of the processes supported by the IOIS?

In the face of conflicts that may exist between competing tourism enterprises within the same destination, the DMO has a critical role to play through its IOIS: that of consolidator, leader and coordinator in order to reconcile divergences and present a global image of the destination [1]. Previous studies [13, 34] found that stakeholders held different perceptions about the role of IOISs and the metrics that need to be used. This case study integrates those different perspectives into a single framework taking into account the vision of its multiple stakeholders. This was made possible using a process-based rather than a variance-based evaluation approach, since a process-based evaluation allows to measure the effects of the IOIS at the level of intermediate processes and functionalities. Furthermore, this evaluation approach offers a framework for DMOs to nurture a collaborative culture [13]: each stakeholder may express his expectations regarding the processes he is using, following which the DMO will request a contribution from those stakeholders. While the IOIS effects and performance indicators are highly contextualized, this process-based evaluation approach is replicable to any IOIS. This approach confirms previous literature in focusing attention on the need to evaluate all the IOIS functionalities with a holistic approach [5, 13, 14]. This approach is in line with Sigala recommendations for future research [13]: (1) to include the view of stakeholders in the private sector; (2) to consider attitudes through the dimensions satisfaction and contribution as well as the expected benefits with the dimension expectations regarding the IOIS. As previous research suggests, a high-level integration between stakeholders and interoperable systems is a requisite to a better performance [1, 5, 17]. This case study confirms previous findings showing that the primary goal of the DMO IOIS is still to provide information and to market the destination to potential visitors, and that less attention is given to their internal leadership role of coordination [35]. For example, Tourisme Montréal utilizes its IOIS neither to support the product development role of the destination nor to provide any B2B information to potential investors. It was found that Tourisme Montréal follows the trend identified in the literature in disengaging from the transactional functionalities as a way to give access to suppliers’ own booking engines [5, 13]. As a consequence, this DMO will need to access third parties’ big data to become an adopter of business intelligence and data science [1, 36]. In order to position this IOIS evaluation approach within the business intelligence literature, the following BI definition will be used. “Business intelligence (BI) is a combination of processes, policies, culture, and technologies for gathering, manipulating, storing, and analyzing data collected from internal and external sources, in order to communicate information, create knowledge, and inform decision making. BI helps report business performance, uncover new business opportunities, and make better business decision regarding competitors, suppliers, customers, financial issues, strategic issues, products and services” [37]. As the resource based-view theory of the firm was mobilized in the theoretical framework, and as this evaluation approach already involves internal and external stakeholders and is process-based, this evaluation approach offers an evaluation framework usable in the context of BI implementation. However, the adoption of BI initiatives requires more than an integrative IOIS performance evaluation approach; indeed, it requires a complete business model shift to place a greater emphasis on building a collaborative knowledge creation environment [1, 36] and, above all, a significant investment in the technologies that will allow DMOs to access and analyze relevant data on a timely basis.

5 Conclusion

The theoretical contribution of this article is based on four points. The first point is to establish a link between the effects of IOISs on processes and the performance indicators of these processes in an inter-organizational context. By evaluating the effects of the IS on processes and not the impact of ISs directly on the overall performance of organizations, the approach avoids the influence of exogenous variables that influence the DMO’s performance. In a context where the literature attests to the difficulties in explaining the effect of IOISs on participating organizations, this research makes it possible to identify and define relevant dimensions: the effects of IOISs and user expectations, satisfaction and contribution.

The second point shows how the methodology used makes it possible to avoid the anchoring bias of an ex post evaluation. The proposed inductive approach avoids the subjectivity of an ex post evaluation that aims to define the level of performance of the IOIS in relation to the objectives pursued by the organization [18] or in relation to objectives determined during an ex ante evaluation. Since there is no potential value, the assessment of the value achieved is based on user satisfaction and the effects of the IS on processes, thereby avoiding anchoring bias.

The third point highlights the multidimensional and contextual aspects of the performance indicators used. By using the performance prism [24], the evaluation approach used takes into account the complexity and the multidimensional and contextual aspect of the concept of performance and does not reduce it to quantitative and financial aspects. This approach considers the expectations of the various stakeholders and their influence on the concept of process performance through their satisfaction; it focuses on the contribution of business partners, who can also benefit from the value of the IOIS when they are not directly part of the DMO [25]. The approach combines within the same evaluation tool individual and organizational indicators of the impact of IOIS on the performance of DMO processes.

The fourth point indicates that the contribution and satisfaction components make it possible to integrate, in one and the same tool, the concepts of voluntary use by external stakeholders as well as that of mandatory use by stakeholders internal to the DMO, in an inter-organizational context. The contribution component contains quantitative indicators (frequency and duration of use) and qualitative indicators (nature and reason for use). The contribution component also includes dimensions specific to IOIS. For example, the quality of inter-organizational relationships may motivate the use of IOISs by SMEs, which means that some stakeholders use IOIS functionalities for no other benefit than to participate in and maintain a close relationship with DMO initiatives.

While this research contributes to improving the understanding and evaluation of the impact of IOISs on the performance of DMO processes, and despite the rigor of the cross-sectional research design, this research has limitations. These limitations stem from choices relating to the interpretive epistemological position, convenience sampling (the study of a single case with nested cases), and the collection and analysis method. Data collected over a single period of time in semi-structured interviews reflect the respondents’ point of view, which exposes them to cognitive bias. Data collection, for external stakeholders, was conducted with only one person per organization. The results of the qualitative analysis of the data are essentially based on the researcher’s interpretation despite the precautions taken to ensure transferability: the creation of a structured database; the use of reflective tools such as the logbook and memos; the validation of certain results with key respondents; and an audit of the coding.

The current state of knowledge justifies the continuation of work on the evaluation of IOIS performance. An interesting approach would be to conduct similar research, but this time with one or more rural DMOs IOIS. Another avenue to explore would be to conduct a longitudinal study of the evaluation of the performance of the IOIS of a DMO in order to have a perspective of its medium- and long-term impact on stakeholders.