This research has made it possible to demonstrate the usefulness of the theoretical framework developed for evaluating the performance of an IOIS and, to propose an approach for evaluating the performance of an IOIS. Figure 2 below illustrates how the dimensions of the performance of the DMO’s IOIS differ according to the internal and external stakeholders involved (answer to specific research question #1).
4.1 Dimensions of IOIS Performance for Internal Stakeholders
The top of Fig. 2 shows the dimensions and performance indicators of the DMO’s IOIS that relate to its internal stakeholders: (1) the capabilities of the DMO and its employees, (2) the expectations of DMO employees with respect to the processes supported, or not, by its IOIS, (3) the automation, information and transformation effects of the IOIS on DMO processes, (4) the satisfaction of DMO employees with the IOIS, and (5) the contribution of employees to the DMO’s IOIS.
Capacities of the DMO and Its Employees.
This first dimension reflects the necessary alignment of the DMO’s strategies with the processes supported by the IOIS, its capabilities as a DMO and the individual capacities of its employees. This alignment provides an understanding of how the DMO can articulate its strategies and the processes supported by its IOIS toward the satisfaction of its employees and in accordance with their expectations and those of the DMO. DMO’s capacities include (1) the capacities of the DMO’s technology department, (2) the individual technological capacities of the employees, (3) the complementary organizational resources (human resources, material and financial resources, training provided) of DMO, (4) the strategic directions of the DMO, and (5) the characteristics of the IOIS and its functionalities. Although the use of the IOIS is mandatory, this does not mean that it is optimal in all departments and for all automated processes.
For example, a difference in the training provided on IOIS functionalities was at the origin of significant variations between departments.
Internal Stakeholder Expectations.
The tool provides measures of internal stakeholder expectations that relate to the time required to complete a process and the costs involved. Internal stakeholders are concerned about improving the quality of the system: “Now all departments are ready for a new IS.” This includes ease of use and access, speed of response and execution, intuitive navigation “like Google,” flexibility of queries, visual appearance and formatting to suit needs. At the same time, they seek to improve the quality of information, based on the criteria real time, up-to-dateness, reliability and completeness: “to become a nerve center of information.” Some internal stakeholders are in favor of integrating functionalities that allow the IOIS to be adapted to relational marketing, to promote the integration of technologies with one another: “a single platform for the entire destination.” Participants also mentioned the need to make better use of all the functionalities of the IOIS and the need to improve the quality of the IS service to support users.
Processes Supported by the IOIS.
In the process of evaluating an IOIS, it is necessary to understand its impact on process performance. To this end, this dimension lists all the processes supported by the IOIS. Throughout the interviews, details are provided regarding the level of automation of all the tasks that make up these management and operational processes.
Effects of the IOIS, Satisfaction, Contribution and Performance Indicators.
Automational, informational and transformational effects are considered when evaluating the IOIS by the internal stakeholders. First, the automational effects of the IOIS are related to the quality of the system, its navigation, stability, processing speed, response time, “just a click away” and remote accessibility. The DMO’s employees mentioned automational effects such as loss of autonomy, work overload, “more steps than before” and the complexity of tasks to be performed in the system. The fact that the expectations of a certain group of users had not been considered during implementation explains these adverse effects. It should be also noted that some sub-processes may be supported differently by the IOIS and cause opposite impacts.
Second, the informational effects of the IOIS relate to the quality of the information and the system’s accuracy, updating, presentation, level of centralization and shareability, and relevance. With regard to the respondents’ responses, the measurement of the time spent carrying out a process and the costs this entails must be included in the performance indicators as they influence user satisfaction. Risk management related to information ownership is reduced when vendors are able to update information remotely in the IOIS. For the DMO, such quasi-simultaneous access to this information means that it can attain ownership of the information and, submit an estimate to potential customers at great speed. Thirdly, transformational effects occur immediately after a new IOIS implementation or new functionality is implemented, since a re-engineering process is underway. This is the case when certain processes are outsourced to DMO members, such as the updating of tourism establishment information and the submission process of group quotes. Some DMO departments can become more autonomous in carrying out certain processes, as was the case with the sending out of the newsletter to Tourisme Montréal members. They no longer had to wait for the communications department to extract information from the database, as a result of which members’ newsletters were issued more frequently, and with up to date information: “awesome, more interesting, motivating to develop these tools; real time marketing.” Transformational effects therefore also have an impact on the perceived value to external stakeholders. Finally, process re-engineering can change the level of autonomy of managers versus coordinators, due to their different abilities to work on a new system: “It’s reassuring to know that it’s available to everyone and that it isn’t just me who knows how to fly the airplane.” However, these transformational effects are expected to be temporary as “data volume grows.”
Contributions Expected from Employees.
This dimension of the tool measures the extent to which DMO employees are involved in using the full potential of the IOIS, such as the constant suggesting of improvements, their regular training on the database, and their possible actions as ambassadors of the online destination: “super users.”
Performance Measures of the Processes Supported by IOIS.
The analysis shows that the indicators used by DMOs measure performance at a global level and are not always related to the effects of the IOIS on processes. For example, at Tourisme Montréal, the client satisfaction rate was measured for the entire destination and does not reflect on the performance of services offered through the DMO’s IOIS. Nevertheless, the sales department had some relevant performance indicators at the process level, such as: the number of leads generated, the number of follow-ups and development actions carried out, the number of coordinated tracking visits, and the number of newsletters sent. It has proven useful to be able to calibrate the costs related to the execution of certain processes according to the time spent on an action. This would be a good indicator of the need to automate or transform certain processes supported, or not, by the IOIS. However, the process/time/cost indicators are missing in this case study.
Furthermore, in this case study, the impact of the IOIS on the overall performance of the DMO could not be assessed due to the absence of performance measurement indicators that link process efficiency (process-level performance indicators) to the overall performance indicators of the DMO. This finding confirms that the effects of technologies on processes do not necessarily correspond to the performance indicators used by process managers .
4.2 Dimensions of IOIS Performance for External Stakeholders
At the bottom of Fig. 2 are dimensions of IOIS performance that relate to external stakeholders, in other words, DMO member organizations.
Capacities of DMO Member Organizations and Their Use of IOIS Functionalities.
As with the capacities of the DMO and its employees, this first dimension reflects the necessary alignment of the DMO’s strategies with the processes supported by its IOIS and the capacities of DMO member organizations. This alignment provides an understanding of how the DMO can gear its strategies and processes supported by its IOIS toward the satisfaction of external stakeholders, in accordance with its own expectations. The capacities of member organizations include their own resources as well as those of their brand’s head offices. It is the availability of these resources that conditions to what extent these organizations really need the DMO’s IOIS.
Effects of IOIS on the Processes, Satisfaction and Expectations of External Stakeholders.
There are, to begin, the IOIS automational effects, which can be measured for external stakeholders in terms of the quality of the system, ease of navigation, ergonomics, speed and interactivity of the system, and flexible formatting of the reports to suit needs. External user satisfaction is positive if the value of the previous measurement indicators exceeds their expectations. It will be negative if not.
The expectations of the external stakeholders can guide the DMO IOIS deployment strategies. Some external stakeholders use IOIS functionalities out of solidarity with the DMO rather than to respond to a real need. When evaluating the IOIS at the process performance level, and hence not only at the level of solidarity use, it became obvious that the individual hotel reservation process was cumbersome, complex and caused errors and loss of time for its users:: “I’m all behind Tourisme Montréal; but it’s not in any way a booking engine, it’s for promotion.” External stakeholders do not all attach the same importance to process automation seeing that they do not all use them in the same way. Expectations for automation are high among hoteliers.
Secondly, the indicators for measuring the informational effects of the IOIS for external stakeholders focus on the quality of the information, in other words, its timeliness, completeness and relevance of its content, timing and conciseness. The sometimes-contradictory interests of these external stakeholders appeared in terms of informational effects because of their different information needs: “long stays markets,” “deluxe leisure markets,” “increased paid visibility” or an “excess of information; we don’t have time to stroll around in members’ extranets.”
Finally, transformational effects are rarely encountered by external stakeholders, as the study revealed, except for the individual reservations process that demanded considerable adjustments on behalf of the hotels.
External Stakeholder Contribution.
For each expectation of the external stakeholders toward the IOIS, a counterpart is expected by the DMO, in the form of a contribution toward the IOIS and/or a voluntary participation. This is a key dimension of the performance prism. All uses of the IOIS by the external stakeholders are voluntary. One member qualifies his relationship with the DMO as “intuitive” and has the impression that he isn’t making “enough use of it due to a lack of time.” Indicators to measure the contribution of external stakeholders to the DMO can therefore be, the frequency of use of the IOIS, the overall value of these uses (member fees, commissions on individual and group bookings, promotional placements, entrance to networking activities, etc.); the average value per transaction; the frequency of suggestions for improvement of the IOIS; and the perceived value of the services offered by the DMO’s IOIS.
4.3 An Approach to Evaluate the Performance of a DMO’s IOIS
The process-based evaluation approach used in this case study brings together all the needs and expectations of external and internal stakeholders by linking them to processes supported by the IOIS (answer to specific research question #2). The process-based approach was found to be the most likely to reflect the mechanisms by which the IOIS could affect the performance of the DMO: the exploitation of the IOIS in the operational processes of the DMO produces automational, informational and transformational effects which, in turn, affect the performance indicators at the level of the operational processes used by the stakeholders. These processes performance indicators have an influence on the overall DMO performance.
If the evaluation process reveals that the IOIS has little or no impact on the overall performance indicators used by the DMO managers in managing their processes, the following questions should be asked:
Is the IOIS used appropriate for the DMO? Do its functionalities meet the expectations of the stakeholders? Are the key processes supported by the system?
Do the DMO’s IS strategies meet the needs of the stakeholders?
Is the implemented IOIS being exploited to its full potential?
Have investments been made in the “complementary assets” (DMO’s capabilities) that enable the development of the IOIS? Does the capacity of user tourism organizations allow the IOIS to support the processes?
Do the performance indicators used by the DMO truly reflect its organizational performance in all its aspects and capture the efficiency of the processes supported by the IOIS?
In the face of conflicts that may exist between competing tourism enterprises within the same destination, the DMO has a critical role to play through its IOIS: that of consolidator, leader and coordinator in order to reconcile divergences and present a global image of the destination . Previous studies [13, 34] found that stakeholders held different perceptions about the role of IOISs and the metrics that need to be used. This case study integrates those different perspectives into a single framework taking into account the vision of its multiple stakeholders. This was made possible using a process-based rather than a variance-based evaluation approach, since a process-based evaluation allows to measure the effects of the IOIS at the level of intermediate processes and functionalities. Furthermore, this evaluation approach offers a framework for DMOs to nurture a collaborative culture : each stakeholder may express his expectations regarding the processes he is using, following which the DMO will request a contribution from those stakeholders. While the IOIS effects and performance indicators are highly contextualized, this process-based evaluation approach is replicable to any IOIS. This approach confirms previous literature in focusing attention on the need to evaluate all the IOIS functionalities with a holistic approach [5, 13, 14]. This approach is in line with Sigala recommendations for future research : (1) to include the view of stakeholders in the private sector; (2) to consider attitudes through the dimensions satisfaction and contribution as well as the expected benefits with the dimension expectations regarding the IOIS. As previous research suggests, a high-level integration between stakeholders and interoperable systems is a requisite to a better performance [1, 5, 17]. This case study confirms previous findings showing that the primary goal of the DMO IOIS is still to provide information and to market the destination to potential visitors, and that less attention is given to their internal leadership role of coordination . For example, Tourisme Montréal utilizes its IOIS neither to support the product development role of the destination nor to provide any B2B information to potential investors. It was found that Tourisme Montréal follows the trend identified in the literature in disengaging from the transactional functionalities as a way to give access to suppliers’ own booking engines [5, 13]. As a consequence, this DMO will need to access third parties’ big data to become an adopter of business intelligence and data science [1, 36]. In order to position this IOIS evaluation approach within the business intelligence literature, the following BI definition will be used. “Business intelligence (BI) is a combination of processes, policies, culture, and technologies for gathering, manipulating, storing, and analyzing data collected from internal and external sources, in order to communicate information, create knowledge, and inform decision making. BI helps report business performance, uncover new business opportunities, and make better business decision regarding competitors, suppliers, customers, financial issues, strategic issues, products and services” . As the resource based-view theory of the firm was mobilized in the theoretical framework, and as this evaluation approach already involves internal and external stakeholders and is process-based, this evaluation approach offers an evaluation framework usable in the context of BI implementation. However, the adoption of BI initiatives requires more than an integrative IOIS performance evaluation approach; indeed, it requires a complete business model shift to place a greater emphasis on building a collaborative knowledge creation environment [1, 36] and, above all, a significant investment in the technologies that will allow DMOs to access and analyze relevant data on a timely basis.