Keywords

1 Introduction

Employers have been placing an increasing emphasis on collaborative problem-solving (CPS) skills [6]. CPS is a non-routine skill employed to solve various complex and information-insufficient problems [3]. In other words, CPS is not a domain-specific routine skill, but a transversal competence that differs from reasoning and working memory. Given that the problems with which people deal in daily work and life are usually complex, not all the information needed to solve these problems is usually available. Additionally, in such situations, multiple goals have to be considered at the outset, and problems cannot be solved solely by reasoning and working memory, but have to be approached by people working collaboratively and using various strategies to gather information. As a result of recent developments in web technologies, online CPS has become a more important means for information sharing and communication [4]. Research has found that the use of web technologies, such as e-mail, videoconferencing, instant messaging, and so on, for virtual collaboration is one of key skills that will be necessary in the workplaces of the future [2]. Thus, web-based CPS has received considerable attention in educational programs and is now part of large-scale educational assessments internationally. For instance, Organisation for Economic Co-operation and Development (OECD) assesses CPS skills related to primary development in the Program for International Student Assessment [PISA]. PISA 2015 defined CPS competency as “the capacity of an individual to effectively engage in a process whereby agents attempt to solve a problem by sharing the understanding and effort required to come to a solution and pooling their knowledge, skills, and efforts to reach that solution” [6].

According to the aforementioned characteristics of CPS, it is important to develop the ability to apply CPS skills to web-based technological scenarios. Consequently, the purpose of this study was to develop a CPS system involving computer agents and web technology to train learners and to assess their CPS skills in building teams, communicating information, sharing knowledge, and developing strategies to solve problems. The development of a CPS system involves careful consideration of several issues. For instance, systems allow learners to collaborate with computer agents or humans. Questions about which aspects of the CPS abilities of learners will be particularly important for collaborating with computer agents and other learners must be addressed when designing training and assessment programs. Few studies have compared the advantages of the collaborative mode involving learner-on-computer agent (LCA mode) with those of the mode involving learner-on-learner (LL mode) in terms of CPS performance.

In recent years, an increasing number of studies have examined computer-supported approaches toward collaborative learning. For example, some studies have focused on developing systems to approach open assignments by supporting collaboration, whereas others have adopted an interactive whiteboard technology to facilitate learners’ interaction and collaboration to help them actively solve problems [5]. Although these approaches have had some good results, Researchers found that learners were not satisfied with their team members, and each felt that he or she made more effort than the other team members did, despite the fact that the skills of learners increased more with CPS than with traditional learning [8]. That is, learners evaluated the efforts and contributions of others in the problem-solving activities while participating in the collaborative process, which thus elicited both cooperative and competitive behaviors [7]. Findings such as these led other researchers to conclude that support of collaborative learning activities and procedures requires specialized tools instead of standard, web-based technologies.

CPS involves collaboration between an individual and other agents during a problem-solving process [6]. These agents may be computer-simulated participants or humans. Many systems using computer agents for training or evaluating in different fields have been applied to overcome difficulties related to the aforementioned competitive behaviors; these include the use of computer agents to collaboratively find solutions [9] and the development of an inferential problem-solving system [1]. Again, additional and more in-depth direct comparisons of the effects of different collaborative modes on learners’ CPS performance are needed. The results of such investigations would be particularly useful for improving a computer agent-based CPS system. Consequently, the goals of this study were as follows: (1) to develop a CPS system using web technology and computer agents and (2) to understand the advantages and disadvantages of different collaborative modes in terms of learners’ CPS performance.

2 Method

2.1 Development of the CPS System

This study developed a CPS system with web technology and computer agents to train and assess CPS skills. Figure 1 shows the interface of the CPS system.

Fig. 1.
figure 1

The CPS system interface presents eight problems

Full size image

The design elements of task characteristics, the problem scenarios, the medium, and the composition of teams were considered in the process of developing the present system. With regard to task characteristics, all tasks in this study were related to learners’ daily work and life, and learners and computer agents had to assume different responsibilities to solve problems. As mentioned above, problems involved tasks that could plausibly appear in the daily lives of learners. In terms of medium, learners were provided with sufficient information to work with computer agents, and were asked to determine the appropriateness of strategies offered by such agents. With respect to team composition, learners worked with computer agents and were required to play different roles in different scenarios.

The problem-solving tasks were designed to allow guidance by the system. Learners followed the system’s instructions and completed each task in 10–15 min. Four tasks were used in the present study: hanging shelves, using a microwave oven, defusing a bomb, and arranging a bedroom. Figure 2 shows the interfaces for the problem involving using a microwave oven, in which participants collaborated to prepare a meal using this device and applied related scientific knowledge that was provided, such as heating method, power, and microwave penetration. In the task involving arranging a bedroom, participants used knowledge about ergonomics and the interior design materials provided to arrange furniture in a bedroom. Hanging shelves involved discussion among participants about how to attach shelves to the surface of the ceramic tiles in a bathroom. In the bomb defusing task, participants acted as police officers and used selected tools to apply exactly four gallons of water on a sensor to defuse a bomb. The reasoning method and mathematical principles of leverage were explained to help learners collaboratively generate strategies and determine the best solution to each problem.

Fig. 2.
figure 2

Interfaces in the problem involving use of a microwave oven

Full size image

CPS performance was rated in terms of the following three skills: shared understanding, problem solving, and establishing and maintaining team organization [6]. For example, the task of building shelves involved 13 questions related to shared understanding, 19 related to taking appropriate action to solve the problem, and 7 related to establishing and maintaining team organization during the CPS process. Students answered these questions based on their interactions during the collaborative problem-solving tasks.

2.2 Experiments

Participants.

Sixty-four college students with an academic background in technology participated in this study. Of these, 36 participants were male, and 28 were female; the age of participants ranged from 18 to 22 years (Mean = 20.1 yr, SD = 1.2). None of the participants had used the CPS system previously.

Design and Procedure.

This study examined the effects of different collaborative modes on learners’ CPS performance using a CPS system. The collaborative mode was a within-subject variable. Half of the participants were randomly assigned to the LL mode first and then to the LCA mode, and the remaining participants were assigned to the LCA mode first and then to LL mode. Figure 3 shows the procedure of the experiment.

Fig. 3.
figure 3

Experimental procedure

Full size image

Participants were invited to a classroom, where they received a brief explanation of the purpose and the procedure of the study. Then, participants used two of four problems to practice the process of collaborating in LL and LCA modes and to become familiar with the CPS system interface in different collaborative modes. After practice, participants took a 2-min rest, and then started to solve the other two problems in either LL or LCA mode. In LCA mode, each participant used a personal computer with a 17” monitor to independently complete the CPS task. In the LL mode, two participants viewed a same monitor and communicated with each other face to face. One of the two participants was responsible for entering their joint responses to the CPS system’s questions. The CPS system recorded all participants’ responses and calculated CPS performance scores. Each participant took about 30 min to finish the experiment.

3 Results

In this study, analysis of variance (ANOVA) was used to analyze whether collaborative mode affected CPS performance, and t-tests were used to compare the effects of the two different collaborative modes on learners’ evaluation of the CPS system. Table 1 shows participants’ CPS performance scores in two different collaborative modes and Table 2 shows the ANOVA analysis.

Table 1. Participants’ CPS performance scores in LL mode and in LCA mode
Full size table
Table 2. ANOVA for CPS performance scores
Full size table

3.1 Effects of Cooperative Modes, Gender, and Learning Styles on Learners’ CPS Performance

Table 1 presents participants’ CPS performance scores (mean and standard deviation) by collaborative mode. The ANOVA revealed that the collaborative mode significantly affected participants’ CPS performance scores (F(1,54) = 24.49, p < .01, η 2p  = .31). Participants’ CPS performance scores in the LCA mode (M = 79.19, SD = 8.88) were significantly higher than those in the LL mode (M = 77.75, SD = 5.63), with an effect size of r = .10.

Participants’ system performance evaluation on two different collaborative modes. The mean evaluation score in the LL mode was 3.15 (SD = .37), and it was 2.97 (SD = .42) in the LCA mode. The mean score on each item related to collaborative mode was higher than the overall mean (Value = 2). The t-test revealed that the overall mean score for the LL mode was significantly higher than that for the LCA mode (t(63) = 3.73, p = <.01), with an effect size of r = .30.

4 Discussion and Conclusions

Given changes in the professional competencies needed for the future, countries have devoted considerable attention to training youth in CPS skills to increase their ability to solve problems that cross traditional disciplinary boundaries [3]. This study developed a CPS system as a tool to achieve the educational goals mentioned above. CPS training can involve working with a computer agent or with a real person. This study investigated the differences in learners’ CPS skills (knowledge sharing, problem solving, and team organization) using two different collaborative modes, the LL mode and the LCA mode.

The results of this study showed that CPS performance differed significantly by collaborative mode. According to previous research, learners who engaged in LL mode were not satisfied with their team members, feeling that they expended more effort than others did. That is, learners evaluated the efforts and contributions made by others while they were collaborating with them, thus simultaneously engaging in both cooperative and competitive behaviors [7]. In this study, participants were asked to solve the problems within a limited period of time by following system’s instructions and using the information provided. However, learners need more time than computer agents to communicate with one another, build trust, and establish and maintain organization [7]. These observations might explain why participants performed significantly better in the LCA than in the LL mode. Although we found a significant difference in participants’ CPS performance between the two collaborative modes, the effect size was small (r = .10); hence, the LL mode might potentially be useful when carefully applied in appropriate situations. For example, the LL mode could be used to train CPS skills when students collaborate with a teacher instead of with other learners. As the relationship between teacher and learners is not competitive, competition between students may be avoided. Furthermore, teachers would be more familiar with the content of the problem and would have effective collaborative and communication skills. Thus, it would possible for them to assist learners in communicating, sharing knowledge, and establishing and maintaining organization. In contrast, the LCA mode is more applicable to assessing learners’ CPS performance skills. The LCA mode could avoid competitive behaviors, save time that would otherwise be devoted to interpersonal communication, build trust, and develop an organized approach involving other learners, leading to more accurate and efficient performance. Moreover, the use of the LCA mode allows greater flexibility with regard to practice times, enabling learners to practice CPS skills themselves using the STEM-based CPS system with a computer agent at any time and in any place. Consequently, this study identified an optimal mode for training CPS skills.

There are some limitations. First, as the sample size of this study consisted of only 64 participants, significant findings should be interpreted with caution. Moreover, the CPS system used in this study was not well developed and requires numerous improvements. However, our results can help researchers to identify areas of improvement for a CPS system, and might inspire other attempts to design CPS-related systems. Future research should involve longer-term studies that include more different collaborative scenarios, a larger sample size, and a wider variety of measures. Second, the problems used in this study were well structured. It would be interesting to examine learners’ transversal competence in addressing poorly structured problems that did not have correct answers to assess differences in learners’ CPS performance and collaborative behavior in response to problems with different levels of difficulty.