Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

Interdisciplinary collaboration is seen as a means to solve complex problems that surpass disciplinary boundaries. By combining methods from different disciplines advances can be achieved that were previously inaccessible. In the Cluster of Excellence “Integrative Production Technology for High-Wage Countries” — a research cluster with over 180 staff members from various disciplines — interdisciplinary collaboration is applied intensively. In order to support this collaboration process, we set up a scientific cooperation portal that supports the researchers in various ways.

The web-based portal offers functionality that addresses barriers and benefits of scientific interdisciplinary collaboration [1, 2]. As an example, a shared file-space that can be easily accessed from the individual institutes is provided. This file-space supports the researchers as it provides a secure storage location of data that needs to be shared across institutional boundaries. Version control makes sure changes to files will never get lost.

The framework used to create such a portal is the open-source software LiferayFootnote 1. Liferay is a portal solution that comes with several out-of-the-box features that are helpful for scientific collaboration (e.g. the mentioned file-space). Our portal started using the Liferay 6.1 version. In this version some users reported having difficulties in using the portal. Liferay 6.2 comes with several bug fixes and included new features such as:

  • Responsive design: The design responds flexibly to changing screen sizes. So it can be used on desktop PCs as well as on mobile devices, like smartphones or tablets. Designers and developers can use a preview feature to test their work on different screen resolutions.

  • Several User Interface Refinements: Beside the general design, also the control elements were recreated. Additionally, the underlying web technologies were updated to provide up-to-date tools for developers.

  • Improved Content Management: New functionality was added to manage and structure portal contents, e.g., folders to manage content articles, dynamic data structures to define own content types, and a recycle bin to restore erroneously deleted contents.

For further details about Liferay 6.2 improvements, refer to the official Liferay blogs [3, 4].

These features promise an improvement in usability, but changing the user interface comes also with switching-costs based on the adjustment to and adaption of new operational concepts. To legitimate these switching costs, which may lead into refusal by the users to work with our cooperation portal, we have to make sure the system usability was actually improved, so that we can expect a long-term increase of satisfaction and acceptance. For this reason, we conducted a comparative usability study of the two used versions, which will allow us to evaluate the supposed usability improvement but also the expected switching costs.

Fig. 1.
figure 1

The home page of the former Scientific Cooperation Portal using Liferay 6.1. The design was based on the Cluster of Excellence website and dominated by an image slider on top.

Full size image

2 Related Work

The scientific cooperation portal was set up to support inter-organizational technology transfer within the Cluster of Excellence. Therefore, we decided to use modern web technologies and social media approaches, which are suitable for this supporting task [5–7].

Our portal meets several functional requirements we identified before [8]: presentation of comprehensive information on technologies or technological know how, presentation of users and their expertise, technology- and application specific information clustering, communication channels et cetera.

Nevertheless user factors must be considered when developing a social software portal that relates to knowledge management and sharing [9]. In particular using such a software in a work environment is fundamentally different than using it in private usage contexts [10]. Users show different sharing behaviors in private and working contexts often moderated by their personality [11]. Often using a social networking site at work comes with new challenges as missing etiquette [12] and differing usage motivation [13]. Neglecting these requirements can lead to rejection of such a tool [14]. Even when individual differences are regarded, one must focus on slick usability [15] to reduce the amount of added complexity into an already stressful researcher life.

All of these findings show, that a successful portal solution depends on a multitude of factors. For this reason, changes to the user interface may cause unpredictable consequences. Accordingly, the preliminarily described user study is mandatory after we updated our existing portal.

Fig. 2.
figure 2

The home page of the new Scientific Cooperation Portal using Liferay 6.2. The design was based on the Liferay default theme and focused on a very simple and clear structure.

Full size image

3 Method

To get a better understanding which impact the portal update actually has on the perceived system usability, we initially analyzed the general usage of the scientific collaboration portal. In this regard, web analytics data usage was collected to especially determine the features preferred by users. To get this data, we used the open source analytics tool PiwikFootnote 2 between October 2013 and April 2015. Among other data, we measured page views and the time spent on each page.

Afterwards, we conducted the actually intended user study in form of two separate studies, which were recorded using the usability testing tool MoraeFootnote 3. In both studies, we asked six users to perform tasks and answer a follow-up study. The study was a repeated-measure within-subject design, comparing the usability of versions 6.1 (Fig. 1) and 6.2 (Fig. 2) regarding the same six tasks.

3.1 Task Set

The following tasks were to be performed using both Liferay 6.1 (1\(^\mathrm{st}\) user study) and Liferay 6.2 (2\(^\mathrm{nd}\) user study).

  • Profile Details: Please validate your personal details. If you find any missing information, please add the information.

  • User Lookup: Please find person N.N. and send him a personal message.

  • Document Download: Please locate the protocol of the last colloquium of employees. Please download this document.

  • Calendar: Please find out when your next project-meeting will take place.

  • Publication Upload: Your last paper was accepted. Please upload the PDF file from the desktop into its designated location on the server.

  • Community Features: Your project leader has uploaded a document in your project folder. Please add your feedback on his document by giving it a 4-star rating.

After the tasks were completed, a post-experimental survey was conducted. Here, we measured usability using the system-usability scale. We also measured user satisfaction, expected product use and various independent variables for analyses.

4 Results

We analyzed both all utterances of participants as well as quantitative information from the post-experiment questionnaire.

4.1 Description of the Sample

All participants were members of the Cluster of Excellence staff and voluntarily took part in the user studies. For the second study regarding Liferay 6.2 we made sure that all participants already knew the previous portal version. We found that at least a third of our participants uses the portal system more than once a month in both studies. Almost all users have been using the portal for longer than 6 months, with a unique exception in the second study. Most researchers reported to use the portal for information purposes (see Fig. 3). Interestingly, the knowledge management purpose decreased for the second study, while the file-space still is one of the most active areas in our portal.

Fig. 3.
figure 3

Reported purpose of usage. Error bars denote standard errors.

Full size image

4.2 Web Analytics Findings

The web analytics data stated a very clear result: By far, the file-space section was the most-used feature, followed by event information and news. Table 1 gives an overview on relative usage frequency and time by portal section between October 2013 and April 2015. These results also fit the participants’ usage purposes as shown in Fig. 3: Predominantly they use the portal for information search.

Table 1. Relative usage frequence and time by portal section between October 2013 and April 2015.
Full size table

4.3 Qualitative Findings

Most of the negative feedback we recorded was addressing naming of menu points, which were done similarly in both Liferay versions by us. This means that some participants did not find the naming of menu entries intuitive (e.g. filespace instead of downloads). Besides that differing levels of menu-depth caused disorientation. Certain menu-levels only contained one sub-menu. This caused the UI-menu to not reveal the mouse-hover-menu. Users mistook this for an error.

Minor irritations were the usage of an American date format in a German research cluster.

A more delicate problem was the internal server architecture. As our server is protected by allowing only a certain address range to access the server, a VPN connection is required to access it. Problematic was that the server itself was outside this address range. Using a proxy fixed this problem, but caused a browser warning for one user.

Almost all users complained about the lack of a good search function. Interestingly, a search function is included in Liferay 6.2, but was disabled in our setting. This issue could thus easily be fixed.

The document management system was configured for the upload of scientific publications (i.e. additional meta-data). Yet users prefered to upload documents as regular documents. Users expected a meta-data auto-detection from pdf files.

A major concern for some users was the continuity of the portal. Users were wondering, what would happen to the data after the funding period of the portal. Researchers demand a software that extends over project funding and can be integrated into several projects, if they are expected to upload and maintain their personal data.

The feature that caused the most trouble is also the seemingly most simple. The calendar behaves differently from outlook and causes slight annoyances for users that normally use outlook to manage their appointments.

The feature most liked by our participants was the member search. The portal supports looking up other researchers and their photos. This was reported to be extremely helpful in a cluster with over 80 active researchers.

Overall, all participants reported that they like the design of the new version and described it as more intuitive than the previous one.

Fig. 4.
figure 4

Usability evaluation according to the UTAUT items. Error bars denote standard errors.

Full size image
Fig. 5.
figure 5

System usability evaluation according to the SUS items. Error bars denote standard errors.

Full size image

4.4 Comparison of Survey Results

We evaluated the the software using items from the UTAUT, SUS and NPS Scale in both trials. Clear differences can be seen for the items It is easy for me to use efficiently, I find it easy to use and The cluster supports the use. These three items show an improvement even for this relatively small sample size. In contrast, the item It allows me to complete my tasks more quickly showed a negative effect, which we in part attribute to switching-costs (see Fig. 4).

The System Usability Scale [16] showed an improvement in most measured variables. Variables were measured on a six-point Likert scale and showed an increase of about one standard deviation for most items (see Fig. 5). In particular inconsistencies in the system were removed and people could see other users performing quickly in the system.

The Net Promoter Score [17] increased from 36 to 62 points (see Fig. 6). Beyond this it showed a large reduction in variance, leaving a more homogeneous picture for the second evaluation.

5 Discussion

The usage of “yet another” software for busy researchers must be planned well [14]. When ease of use and usefulness of such a software are not sufficient, social software is not being used. This problem is amplified by the social nature of social portals. A threshold of users is required to create an agile social community, but if the first users are dissatisfied no social interaction will ever take place.

Fig. 6.
figure 6

System usability evaluation according to the Net Promoter Score. Error bars denote standard errors.

Full size image

Our research especially showed, that migrating a software to a complete new user interface may come as a drawback of losing users that already have come accustomed to the user interface of the older version. In our specific case we can argue, that a migration from Liferay 6.1 and 6.2 can be conducted without the loss of users, because the new version improves the usability regardless of switching costs. We also attribute the success in migration to the integration of users in the upgrade process. Menus were renamed according to users expectations and the participants took part in communicating the benefits of the new platform.

Overall, Liferay shows the potential to be used as an open-source research management system when tweaked accordingly. Some of the features, that are a problem for users, are the non-typical calendar and the non-intuitive sub-community management. For the latter, the mental model of how to access a sub-community was not immediately clear and thus needs to be improved in the user interface.

5.1 Limitations

Our study was conducted as an exploratory study to both measure and steer the usability and usefulness of our scientific cooperation portal. The relatively small sample size prevent more in depth statistical analyses, but is still helpful in understanding a migration process.

6 Outlook and Future Work

Because of the small quantity of participants, we plan to re-validate our usability findings based on a bigger survey within the Cluster of Excellence. In spring 2016, the new portal version is in use for more than a year, which will allow a new examination without the interference of switching-costs. Overall we plan to continuously monitor the usability of our portal in order to ensure that future migrations can be equally successful.