Keywords

1 Introduction

Nowadays, people use online software applications to support sharing of thoughts regardless of space and time constraints. Platforms for online forums can play an important role in supporting debates, but existing software environments sometimes fail in providing features and design choices to promote more informed discussions. In this context, participants may invest a high amount of effort without achieving a mutually acceptable outcome. This problem is aggravated in free-for-all forums such as those from major newspapers websites or social networks, where individual self-expression prevails, leading to monologues and flaming [7, 8]. These problems may lead to people avoiding to participate in discussions, even when the outcome may affect their lives. Although recent literature proposes methods and tools for online deliberation [14], there is still a need to further study their effects and how they may foster better discussions in which people are willing to participate.

Deliberation is a central skill to exercise respect for different perspectives and still be able to collaborate towards a common objective. Practice on deliberation in an academic context may prepare future researchers to an active and democratic behavior in their research communities and general society [11]. The challenge is to obtain the adequate software features for academic deliberation support. While a rigid software structure promotes a more focused discussion at the expense of limiting the variety of ideas; a flexible structure may promote unexpected and creative participations at the expense of blurring the discussion focus and avoiding proper deliberation [10].

In this article, we investigate three different deliberation-based platforms by assessing their different characteristics in a specific scenario of a graduate course in Human-Computer Interaction (HCI). The goal is to understand the interaction features that may influence people’s engagement. In our research methodology, we first selected the platforms based on a literature review. We explored Considerit [7], Debate Hub [15] and Trello [1]. The selection of these platforms was guided by their key features. Their choice is justified by the fact that they provide different styles and structures for discussion. First, we conducted a structural analysis to understand the design structure and interaction flow in the platforms.

We then set up their use in the context of a HCI graduate course during a whole semester by involving 17 graduate students and 1 facilitator. The platforms were used as a discussion space for pre-chosen HCI and Philosophy of Science scientific papers. During the semester the students had 9 reading assignments. After the finalization of the course, we collected quantitative usage data obtained by server logs, which provides hints concerning the participants’ engagement and prefered collaboration patterns. This data was an important asset towards understanding which design features might influence engagement, be it in a negative or positive manner.

Furthermore, participants provided qualitative feedback concerning how the platforms influenced their perception of the topics being discussed, as well as their preparation for the a posteriori face-to-face discussion. Participants’ opinions were thoroughly examined to obtain insights that might be useful to inform the design of online discussion platforms. Results indicate that most of the participants were positively affected by the platforms regarding their understanding of the papers. Our findings provide central elements to guide the design of online deliberation software platform to be explored in academic settings.

This paper is organized as follows: Sect. 2 describes the background to the study, Sect. 3 presents the investigation context and the methods used in the analyses; Sect. 4 reports on the results with respect to the structural analysis, quantitative analysis and qualitative analysis regarding the use of the platforms, followed by a discussion on our main findings. Finally, Sect. 5 presents conclusions and suggests further work.

2 Background on Platforms For Discussion

In online discussion forums, participants can either start a new discussion thread by creating a new post or continue an existing thread by replying to others’ posts. Posts in a given thread are linked to each other in a chronological order. Literature has acknowledged the benefits of threaded forum for supporting online discussions and learning. The participants potentially have time to thoroughly thinking before responding and have access to the whole discussion to examine and reflect on the ideas presented. Nevertheless, the threaded online forums might not be the best choice for supporting collaborative processes important in learning situations [6].

Gao et al. [6] synthesized among the constraints of threaded forums: (a) the difficulties of maintaining focus; (b) of promoting interactive dialogues; (c) of synthesizing ideas; and (d) lack of emotional cues. The authors point out other relevant literature mentioning digressions, for example, students posting their own ideas without paying attention or responding to the others’ ideas. They remain in the surface of discussion, seldom going deeper through negotiation of meaning. Furthermore, some researchers acknowledge the problems due to the structure and design features of those software systems, demanding efforts into their design to increase the likelihood of effective discussions. Relying on this fact, we have investigated deliberation platforms, to serve a similar purpose in the context of this study.

Other models of online discussion involve deliberation and debate among people. Online deliberation is defined by Towne and Herbsleb [12] as “a Web-based form of reasoning that gathers and carefully considers options for action and possible consequences of each”. Through the Internet, platforms for online deliberation go beyond gathering information, allowing the exploration, synthesis and critical examination of new knowledge. While a debate is usually an oppositional process where ideas are put forward and defended, or proved wrong, a deliberation assumes that participants propose pieces of an answer to a problem. Thus, it is a collaborative process seeking common understanding and common ground for action. Other authors, as for example, Davies and Chandler [3], explain both deliberation and debate as parts of the same process. They take “deliberation” to denote “thoughtful, careful, or lengthy consideration” by individuals, and “formal discussion and debate” in groups, i.e., as a type of communication among people that is reasoned, purposeful, and interactive.

As shown by Towne and Herbsleb [12], most online deliberation platforms have roots in the Information-Based Information Systems (IBIS), developed in the 70’s. They were transformed into a graphical version in the late 80’s for recording the design rationale of small groups of designers, the graphical IBIS (gIBIS) system, which evolved to the current Compendium open-source software. The knowledge structure graphically represents the topics, issues, questions of fact, positions, arguments, and possible relationship among these. Besides Compendium, several other systems extended the IBIS concepts. Towne and Herbsleb [12] elaborated five design requirements for a deliberation platform to be useful: (a) it must attract contributions, (b) make the deliberation content navigable, (c) have conformance to usability standards, (d) focus on quality content, and (e) promote wide-scale adoption.

Early research has proposed ConsiderIt, a platform where people can publicly deliberate and reflect on others’ thoughts without emphasizing direct discussions. This platform showed its utility to general public measuring the discretionary use in a real web deployment in the context of an election [5, 7]. The probable deliberative disposition of the self-selected users highlighted opportunity for a controlled experiment that revealed how ConsiderIt led to significant changes in standpoint, perceived knowledge, and perceived understanding [11]. In addition, participants’ willingness to include counter-arguments in their statements and to change a standpoint on the basis of new information supports the platform’s potential to increase people’s deliberative skills and attitudes. Our study takes the next step by applying deliberative platforms in a real academic context to facilitate students’ understanding and critical analysis of academic literature.

In this work, we define “academic deliberation” as an exercise of people’s exposure to and processing of diverse information from others; willingness to argue with argument quality, and to participate in the debate of ideas. In this task, participants are challenged or exposed to new and contradicting information or ideas of others. Instead of reaching a common understanding, the aim is to develop critical thought based on their own as well as on the others position regarding the ideas being discussed. As Gao et al. [6], in the concept of “academic deliberation”, we acknowledge the dispositions to: (a) discuss to comprehend and vice-versa, (b) discuss to critique, (c) discuss to construct knowledge, and (d) discuss to share, which address different and interrelated perspectives to learning.

Based on the above discussion, we raised some questions regarding the design and use of deliberation platforms to support academic discussion: How can we better understand online deliberation in certain academic contexts? Do existing platforms address specificities of academic deliberation? How can we better inform the design of online deliberative platforms aimed at involving people in a meaningful, consequential and inclusive way?

This study addressed the three following platforms:

  1. 1.

    ConsiderIt [7] supports public deliberation where users are encouraged to reflect upon the issue by considering trade-offs, as the discussion is framed by arguments featured as pros or cons. People can position themselves in a free scale with extremes in “agree” and “disagree” and contribute with pro or con arguments.

  2. 2.

    Debate Hub [15] supports communities in raising issues; sharing ideas, debating pros and cons, and voting in contributions with a metaphor of thumbs up or down. It allows the community to collectively organize ideas based on their acceptance by its members.

  3. 3.

    Trello [1] graphically organizes information in boards. The boards, in turn, can be divided in lanes containing cards as a unit of information. Even though Trello is mostly used for project management, it has a flexible representation of information, which allowed us to appropriate the platform with specific conventions and color codes to use it as a platform for group deliberation.

3 Study Design

In this section, we present the study context and the involved participants. Afterwards, we report on the methods conducted to collect and analyze data within quantitative and qualitative perspectives.

3.1 Context and Participants

This study was conducted during a semester in an Human-Computer Interaction (HCI) discipline of a Computer Science graduate program at the Institute of Computing, University of Campinas, Brazil. The discipline had 60 h in the semester, organized in 2 classes per week, with 2 h each. It was conducted by a professor with support of an facilitator.

Each week, students were assigned to read and discuss a selected peer-reviewed academic paper about a hot topic in HCI and Philosophy of Science research (e.g., [2, 4, 9, 13]), summing 9 papers in total. They were invited to use one of the platforms to discuss the paper, a week long online activity denominated “warm up”. They were invited to formulate propositions regarding important and/or controversial aspects of the papers; and arguing by means of pro or con arguments. Depending on the platform used, they could provide different forms of feedback to classmates by including: comments (ConsiderIt and Trello); thumbs up/down (Debate Hub); or even reuse classmates’ arguments to compose their stance (ConsiderIt). After a week of the online “warm up” through a platform, the paper was discussed in class in the form of the professor guided debate.

Students used ConsiderIt during 6 weeks, Debate Hub during 4 weeks and Trello during 2 weeks, summing 12 weeks in total. The last week using each platform was dedicated to discuss the platform itself in the light of the previous discussions (platform’s meta-analysis). The participants discussed the way the platform’s design helped or disturbed the deliberations as well as their understanding of the papers. The participants’ feedbacks constituted the main data for our qualitative analysis.

The participants involved 17 graduate students (11 male and 6 female). Most of them (11 in total) were already used to read at least one academic paper per week. Regarding reading strategies, participants are well divided among those who: (a) do not summarize; (b) only highlight the relevant points; and (c) those who make summaries. Only one participant reported to use concept maps. Around half of them do not use online discussion tools, the other half cited discussion forums like those present in Stackoverflow Footnote 1, Slack Footnote 2, and Moodle Footnote 3.

The platforms were briefly presented in class by the facilitator before the first time of their use. The facilitator posted in the Learning Management System the link to the platform and presented brief instructions including deadlines of the activities. From time to time, some additional feedback was included by the facilitator to make clear some of the academic expectations. For example, clarifying that students should include citations and references to substantiate their arguments, and incentivizing them to engage in more difficult tasks, like to elaborate proposals or to use less obvious platform’s features. The participation in online discussion was rewarded with up to 5% of the final mark in the course, depending on the quality and quantity of tasks performed. The use of the platforms during the whole semester produced the interaction data for our different types of analyses.

3.2 Methods

The first step involved to analyze the interaction structure of the platform’s interfaces. To this end, we studied the terms used to denote the different interaction concepts in the three platforms. Our goal was to understand how the interaction is organized in each platform.

The next step involved to examine the usage of the platforms. We explored quantitative and qualitative analyses to comprehend distinct aspects regarding the platforms. This study assumes that by exploring these types of analyses may reveal thorough aspects relevant for deliberation platforms’ design. Our study intended to quantify the interaction among the students in each different platform and specific content discussion. For this purpose, we devised a metric of participants’ engagement by counting all activities (proposals, arguments, comments, endorsements) done by them during each reading assignment, which was obtained through usage logs of each platform.

The endorsement data were slightly different in each platform. In ConsiderIt, we considered an action of endorsement when a participant appropriated (i.e, got behind) another participant’s argument. In Debate Hub, endorsement was counted by means of “thumbs up” or “thumbs down” given by participants on arguments, which means that an argument is flagged as convincing or unconvincing respectively. Lastly, in Trello, we considered the action of adding yourself as a member of a card as an action of endorsement, similar to ConsiderIt’s appropriation concept. We focused on participant’s engagement because we consider it a central aspect in deliberation platforms, and participants’ endorsement behavior, in turn, remains an important aspect of the engagement. We analyzed the activities regarding the reading assignment separately from the platforms meta-analysis.

For qualitative data, we focused on participants’ feedbacks regarding the platforms from the meta-analysis activity. Unlike the normal reading “warm-up”, in the meta-analysis, the proposals were provided by instructors to access students’ theoretical understanding of the previous readings. In this case, the platforms were used as a practical context to exercise students’ design and evaluation skills with regard to aspects that appeared from the papers reading. The following aspects were considered:

  • Human vs. User: As discussed by Bannon [2], a piece of mediating technology can be designed to simply improve productivity, as in the perspective of classical HCI, the “user” perspective. However, the author argues for a new approach that “encompasses a much more challenging territory that includes the goals and activities of people, their values, and the tools and environments that help shape their everyday lives”, the “human” perspective. In our study, for each platform, the participants were provoked with the proposal “This tool is for users instead of humans”.

  • Neutrality: According to Fallman [4], technologies always transform experiences in predicted and unpredicted ways; hence it is not possible for them to actually be neutral: “for every revealing transformation there is a simultaneously concealing transformation of the world, which is given through a technological mediation”. For this reason, “the user needs to care for the mediating technology, which might come to affect both how people behave in certain situations as well as how others perceive them”. For each platform, the participants were asked to position themselves in relation to the following statement: “This tool is not neutral with regard to the discussion”.

  • Change of Perception: Inspired by Traunmueller’s research [13], which investigated visual factors that promote a change of safety perception in urban places, participants were asked to discuss upon the following statement: “This tool alters the perception of understanding of the assigned reading”. The participants were asked to highlight examples and/or design features from each platform to illustrate if and how its use affected his/her perception of the reading.

  • Bugs and Features: In response to the proposals: (a) “This tool can help me” and (b) “This tool is better than the previous one”, the participants were invited to provide feedbacks regarding the platforms’ features, report bugs, in addition to issues they experienced by using the platforms. For instance, by starting from the second platform, the participants answered to proposals in the form “Debate Hub vs. ConsiderIt”. They could compare platforms’ features they had used so far.

We considered these aspects relevant for the design of deliberation platforms, and to the best of our knowledge, they were never analyzed in the literature in the context of deliberation platforms. We manually synthesized the participants’ main arguments based on these aspects looking for understanding the way platforms’ design choices affect their usage and perceptions. When appropriated, we summarized the syntheses of the arguments according to categories that emerged from the participants’ opinions.

4 Results

In this section, we present our findings in the three distinct analyses. We start by analyzing the deliberation platforms, their characteristics, structure and interaction flow. Afterwards, we report on the results regarding usage analyses, which emcompasses a quantitative and a qualitative analysis.

4.1 Structural Analysis

ConsiderIt main user interface (Fig. 1) includes the main interaction elements involved in the socially enhanced personal deliberation. The deliberation category (1 in Fig. 1) groups related proposals. It allows to navigate through proposals in the same group. The proposal and opinion slider (2 in Fig. 1) consist in a statement, its author and details. The slider allows the user to choose in a continuum between “disagree” and “agree”. The user may move a blue “face” icon which varies from unhappy to happy depending on the agreement level. Over the slider there is a pictorial histogram that represents the stance of other users. In another view, this histogram can be used to explore others’ opinions by segments, highlighting the arguments of groups of users with similar opinions.

Fig. 1.
figure 1

ConsiderIt main interface: (1) proposal category; (2) proposal statement and opinion slider with others’ opinion pictorial histogram; (3) new arguments entry in the center and draggable arguments of others in the interface borders; (4) comments to the argument.

Full size image

A opinion may be supported by arguments (3 in Fig. 1). Others’ arguments are presented on the interface borders and can be dragged into a personal argument list in the center. In this list, the platform enables the users to write new pros and cons composed by a succinct summary, and details of the argument that might include evidences backing the argument. Once published, new arguments become available to others. Authors are represented by avatars on the side of the argument balloon. When someone else appropriate the argument, i.e., get behind it, this person’s avatar appears behind the author’s avatar. By clicking on this supporters’ avatar cluster, the platform presents their different stances in the histogram. Users can also post comments (4 in Fig. 1) to an argument. They are displayed within the arguments details when the summary balloon is clicked.

Debate Hub main screen (Fig. 2) shows the main discussion elements. There is an issue or debate title (1 in Fig. 2). It presents a description and a summary of how many views, ideas, participants and votes were posted. At this place, the platform indicates the remaining time to end the discussion. The votes are used to rank the ideas in terms of community acceptance. There is a feature designed to enable users to submit a new idea, as well as to view previous ideas status (2 in Fig. 2). It includes the title, description of an idea and author’s avatar. The thumbs up/down are used to cast a vote claiming the idea as strong or weak. The vote bar indicates the current support for the idea including arguments and votes.

Fig. 2.
figure 2

Debate Hub main screen: (1) issue; (2) submission of a new idea or votation of others’ ideas; (3) addition and vote for/against arguments. Dashbord of computer supported analytics with debate health indicators and alerts (right).

Full size image

The “for” and “against” arguments are divided in two lists under each idea (3 in Fig. 2). They are represented with a title, its description and the author avatar. Thumbs up/down display the number of “convincing”/“unconvincing” votes received by the argument based on clicks on it. The interface affords the addition of new arguments at the bottom of each list. Unlike ConsiderIt, there is no support to propose comments on arguments and the interface do not reveal the voters.

The Debate Hub features computer-supported debate analytics in the form of a debate healthy semaphore (Fig. 2, dashbord on the right) and alerts including hints to moderators. The healthy semaphores indicate key aspects of discussion flow. For instance, if there is balance between types of posts and community participation; it has links to advanced graphic visualisations of debate progress and other usage analysis.

Trello (Fig. 3) organizes information in a visually hierarchical structure. The board (1 in Fig. 3) as the top level element is used to group related proposals. New boards can be created to hold debate on other issues. Inside a board, the lists (2 in Fig. 3) divide the board in subgroups used to represent each proposal in discussion. The list title holds the proposal statement. A new proposal is introduced by creating a new list.

Fig. 3.
figure 3

Trello main screen interface: (1) discussion board; (2) proposal list; (3) con/pro cards; and (4) comments indicator in the card and comment in the activity log (on the bottom right).

Full size image

The user interface element named “card” (3 in Fig. 3) refers to the unit of information inside the list used mainly to hold arguments. Besides its title, a card displays (when the card is opened) details of its description and activity log. The card can be marked with a colored label; the used color convention was: blue colored card to hold the description of the proposal; green or red card to hold pro and con arguments, respectively. Users can become card members to show support for a card. The members are displayed as avatar below card’s title.

Trello affords the posting of comments (4 in Fig. 3) as another type of direct interaction with others’ card. The user interface indicates, in front of the card, the number of comments a card received. All the actions performed by users are logged into activity indicator (right) and are notified to all those interested in a card. Trello has also a plugin that supports vote for a card, indicated by the number of thumbs up on the card, but lacks a way of computing total votes by labels or lists.

To configure this platform for deliberative usage in a way more aligned with the previous platforms, the first list of proposal was used to hold extra usage conventions. The first card (blue) defined 10 textual rules to attribute deliberative semantics and meaning to UI elements and actions, e.g. “A card with con argument must be labeled red”, “Agreement with the argument in a card is expressed by becoming member of it”. Also a card with each color label was given as example.

The platforms use a similar hierarchical approach to organize discussion by employing different terms to designate the involved concepts. In the first level, different terms are used to group the issues to be discussed. In the second level, while ConsiderIt collects “opinions” instead of “proposals”, Debate Hub counts “votes” for “ideas” and Trello has abstract “list” and computes “votes” only in the third level, for “cards”. Table 1 presents our mapping of these interaction concepts in each platform.

Table 1. Terms that designate the interaction concepts in each hierarchy level of the platforms.
Full size table

In addition to the levels treated, and less visible in the user interfaces, each platform defines and manages the working group and website address. ConsiderIt provides a subdomain with specific URL for each community; the Debate Hub allows the creation of “Discussion Group” for each community/project; and Trello employs a similar approach, but name it as a “Team”.

4.2 Usage Analysis

In this section, we present the results for the quantitative analysis followed by the qualitative analysis.

Quantitative Analysis. We examine the participants’ engagement for the three distinct platforms. First, we computed the sum of average contributions from all reading assignments (cf. Fig. 4). The contributions refer to messages related to proposals, pros and cons arguments, comments and endorsements.

Fig. 4.
figure 4

Engagement as the sum of average contributions (proposals, cons, pros, comments and endorsements) using the platforms for paper discussion.

Full size image

Results reveal that the number of proposals remains similar to all platforms. Considerit presents a higher number of cons arguments while Trello presents a higher number of pros arguments, which are represented by green cards. Debate Hub does not have the feature of comments, and we observed a higher number of participants’ comments in Trello. With regard to the Endorsement activity, the results show that participants explored Debate Hub much further than the other platforms, possibly to compensate the lack of comment support. Other way used to workaround this limitation was by citing previous argument’s authors in the new argument text.

We further computed the contributions in the meta-analysis activity for the three platforms (cf. Fig. 5). First, the number of contributions remains higher than for the reading assignments. This may be explained by the fact that, in these activities, the proposals were provided by instructors based on already discussed papers and experienced platform usage. Debate Hub presents the highest number of contributions mostly due to the Endorsement. In this platform, the number of cons arguments increased with respect to the results presented in Fig. 4. Trello presents an increased number of Endorsements compared with Fig. 4. This may be explained by the learning effect, since not all participants understood how to use this feature in the only paper discussed in this platform, but received additional instruction before using it again in the meta-analysis.

Fig. 5.
figure 5

Engagement as the sum of average number of contributions (proposals, cons, pros, comments and endorsements) for the meta-analysis activity by using the platforms.

Full size image

Qualitative Analysis. The qualitative data shed light on design principles and practices that converge or diverge among the platforms. In this section, we present the results for each of the aspects investigated in our study.

Humans vs. Users. The majority of participants disagree with the statement “This platform is for users instead of humans” for all platforms. Among the collected feedbacks, we highlighted the design features that may have led them to perceive it as “for humans”:

“The non binary ‘opinion slider’ reflects human beings [subjectivity], even allowing one person in complete agreement incorporate opposing arguments in his/her opinion”. (Participant 8, regarding ConsiderIt)

“Focus on the ‘health’ of the community [...] as can be seen by the healthy semaphore indicating participation, group awareness and balance” [in the discussion] (Participant 2, regarding Debate Hub)

“It is interesting to think about different types of discussion and the open possibilities” [in reorganizing discussion flow and card meaning] (Participant 3, with respect to Trello)

Participants reported that the ConsiderIt has an “educative design” to promote better discussions by teaching users to develop refined communicational habits. Furthermore, it allows breaking a complex discussion into simpler pieces of argument with a practical and guided interface with examples and hints. Trello, on the other hand, had usage rules that were harder to learn, but participants could collectively review and redefine these rules.

Neutrality. The majority of participants recognized that a deliberation platform cannot be neutral. Most of the feedbacks were applied for all studied platforms. The non-neutrality concerns mostly discussions framed by pro vs. con assertive arguments, and the platforms lack the use of non-verbal communication, as highlighted by a participant.

“Non-neutrality comes from how the platforms mediate the discussion. For instance, by sacrificing non-verbal aspects of a discussion such as body language, voice intonation, and interpersonal distance”. (Participant 4)

Other non-neutrality aspects were more salient in some of the platforms. For instance, in ConsiderIt, the positioning in the histogram, by avatars’ adjusted position, to indicate the user stance. Regardless the inherent non-neutrality, participants understand the value behind such design decision, as follows:

“The exercise of expressing a discussion in terms of propositions is a useful practice to reflect on the topic addressed. The polarized format of the responses, in turn, makes the tool attractive for more pragmatic uses, such as popular consultations”. (Participant 2)

Furthermore, the automated ranking of provided information affects the neutrality since the order of others’ arguments are presented using a custom ranking algorithm. Such aspects were less perceived in Debate Hub debate analytics. Some students also felt the “Disagree” position as less comfortable, and this could marginally favor the “Agree” side. This is reinforced by design hints, which includes the use of red for disagreement and green for agreement. Trello received some appreciation for the more flexible structure allowing more neutral considerations.

As a counterpoint, the neutrality goes further than the platforms design choice. For instance, one participant highlighted that the nature and quality of the content affect the discussion flow, in particular, the phrasing of the propositions.

Change of perception mediated by the platform. Most of the participants agree that by using the platforms after individual reading positively affected their understanding of the papers. Some of them, however, remained indifferent. Among the collected feedbacks, we highlighted arguments that explain how the platforms lead to a change in perception:

“ConsiderIt is very effective in showing the grouping of opinions according to their point of view. [...] We feel encouraged to rethink, reflect and better support [our point of view].” (Participant 10, endorsed by 5 colleagues)

“As we have to present two positions [...] sometimes I felt induced to manifest both for and against [reflecting on trade-offs].” (Participant 17, endorsed by 1 colleague)

After a complete semester of weekly deliberations, participants perceived: (1) arguments based on personal experience or citation induce more reflection; (2) different people may highlight different aspects of the text, and collectively they construct the discussion flow; (3) participants must develop capabilities to craft insights as proposal or evidence grounded arguments to better influence the discussion flow; and (4) the routine use of the platforms may lead to the need of review some points that seemed clear before the online discussion.

Features and Bugs. We compiled the main aspects to compare the platforms into four dimensions. The dimensions emerged from the manual examination of the participants’ contributions in the meta-analysis activities, which are: learning curve, transparency, engagement and bugs (cf. Table 2). The aspects summarized in Table 2 are marked according to the perceived outcome of the discussion: as () positive or () negative, when there was no objections among participants or the majority agreed on it; or as () controversial, when discussions did not reach a clear consensus.

Table 2. Summary of participants’ feedback comparing the platforms regarding learning curve, transparency, engagement and bugs. Each aspect was considered among participants as () positive, () negative or () controversial.
Full size table

The results regarding learning curve tended towards ConsiderIt. It showed a clean visual, and was judged as a relatively simpler interaction structure. Debate Hub appeared mostly as traditional online forums structure, which was appraised by some participants as easier to read the entire discussion thread without extra clicks. Trello was considered more complex, since it demanded extra work to learn the UI usage conventions.

With respect to the transparency, ConsiderIt received positive feedback due to its graphical overview of the group positioning and the explicit visual endorsement. The Debate Hub thumb up and down interface feature for arguments was welcomed, but the participants expected to visualize who cast the votes. Trello presented an endorsement feature as visible as the feature implemented in ConsiderIt. However, the instructor needed to present to the participants how to use the “card member” feature to express endorsement. In this context, the participants judged negative the lack of visual or numerical summary on the proposal. This occurs because votes in Trello are summarized by cards (third level), but not by lists (second level) as the other platforms.

The engagement dimension attracted most of the students’ attention indicating that they perceived its importance for group collaboration. According to the participants, the engagement appeared more natural with ConsiderIt. It supports direct feedbacks through comments in arguments. The Debate Hub enforced deliberation (without comments or references to other’s arguments) appeared more difficult for participants. Direct notification and activity summary of Trello was very appreciated and may explain part of the increase in the number of comments (cf. Fig. 4)

The bugs considered mainly whether the platform presents a mobile responsive layout. Trello as a commercial platform was the most stable. ConsiderIt and Debate Hub presented some interface issues during their use. In ConsiderIt, some participants claimed to lose the text of some arguments, and even reported some bugs that were fixed by developers. In Debate Hub, one participant claimed to be unable to create an user account.

4.3 Discussion

This study proposed to investigate online deliberation in academic contexts by analysing the interaction structure of distinct software platforms, their utilization via analyses that inquired and measured engagement, endorsement and aspects of neutrality and change of perception. Our findings showed a tradeoff between rigid and flexible interaction structures provided by the platforms. We found that some features seemed not helpful for academic deliberation. For example, the identification of the most acceptable proposal from the discussion, via direct opposition of ideas, as featured by votes in Debate Hub. On the other hand, the ranking feature in ConsiderIt by using avatar histogram was capable of contributing for the transparency as a Graphical overview of each participant positioning.

Although the engagement was more prominent in the Debate Hub (Figs. 4 and 5), this was mostly attributed to endorsement. In this case, we thoroughly examined how endorsement differs among the platforms. The anonymous vote in Debate Hub remains the only kind of direct argument feedback using interface features in this platform. It was probably a lesser commitment when compared to getting behind others’ arguments as occurs in ConsiderIt and Trello endorsements, through personal avatars. We can assume that comments and endorsements complement each other as forms of direct feedback among users. For instance, we observed a significant reduction in comments after the participants have learnt how to endorse using the feature of adding membership to card in Trello.

Besides the effects of interaction design on the engagement metrics (Fig. 4), we found that the activity design and the nature of concepts under discussion naturally influence engagement. For example, the engagement level doubles in meta-analysis activities for all platforms (Fig. 5). This is partly because more proposals were provided by instructors, the readings were already discussed in class, and the necessary experience using the platforms were already acquired.

The qualitative analysis highlighted some essential aspects shared by the studied platforms. While all platforms contemplated human valued aspects, each platform manifested it in a different manner. For instance, ConsiderIt presented its continuum opinion slider enriched with others’ opinion histogram; Debate Hub presented its healthy debate semaphore indicator that contributes for the community; and Trello presented its flexibility to accommodate group personalized discussion flow. Although no mediating platform can be neutral, the platforms’ non-neutrality targets the human necessity of collaboration and knowledge sharing.

This research elucidated that the deliberation platforms were capable of mediating a change of perception regarding the understanding of the selected papers, and promoted reflection on the complex concepts involved. Although it occurred in the proposed activities at some degree, the platforms lack aspects relevant to structure the discussions. The most prominent aspects were specific features to deal with the difficulties faced by participants to formulate instigating proposals.

The features reported by the participants’ opinions reveal salient elements behind the design choices that differentiate the platforms and their outcome. We detected that specific features, such as graphical representations, can help users perceiving and understanding the positioning of their peers. This influenced further reflections and the way the participants prepared themselves to participate in the face-to-face discussion. Nevertheless, it required a slightly modification in the activities setup to enable the participants to identify relevant and/or controversial material to formulate proposal for discussion.

This study enabled to detect to which extent the existing platforms address specificities of academic deliberation, but it presents some limitations. To further support online deliberation in this context requires more situated design to elucidate relevant interaction features. We need to help participants to clarify the addressed issues and identify the affected stakeholders. This may reduce difficulties faced by people to formulate good proposals in academic deliberation. In further work, we plan to design, develop and evaluate a platform for academic deliberation, with prospective interested parties.

5 Conclusion

Online discussions require adequate Web-based software application to support people in proposing ideas for action and argumenting regarding the consequences of these options, for a deliberation. Although literature has proposed platforms to this end and studied human behaviour in this context, the design features of software platforms to support argumentation and debate of ideas in academic deliberations remains hardly investigated. In this paper, we conducted a comparative analysis of three online discussion platforms, based on a real academic context of use. Results revealed central aspects and limitations to guide the design of online deliberation software systems. We found that the discussion quality is highly dependent on the design choices the participants encounter for elaboration of well directed propositions. Further research involves a thorough characterization of the nature and values of academic discussion and the co-design and development of a situated-interaction platform.