The empirical work was conducted following an AR methodology. The purpose of an AR methodology is to influence or change some aspect of whatever is the focus of the research (Robson & McCartan, 2016), trying to improve a certain aspect of the studied phenomenon, in this case the assessment of wiki contributions. According to Oates (2006), the main features of this methodology were as follows:
Concentration on practical issues: an authentic assessment of collaborative work in a wiki-environment.
An iterative cycle of plan-act-reflect: several iterations were performed following the scheme shown in Fig. 1. The first iteration comprised a quantitative analysis through SMW (a tool introduced in Literature review section), and a second comprised the application of the proposed methodological framework in this work and AMW. In some cases, additional iterations were applied to refine indicators or assess new skills.
An emphasis on change: improving previous experiments assessing generic skills based only on quantitative data.
Collaboration with practitioners: people working in the situation under study, i.e. students working in their wiki assignments.
Multiple data generation methods: both quantitative and qualitative data.
Action outcomes plus research outcomes. AR outcomes relate to both “action” and “research”:
Action: practical achievements in the problem situation, i.e. the assessment of students’ skills by their wiki contributions.
Research: learning about the processes of problem solving and action in a situation, i.e. the refinement of indicators using quantitative and qualitative data and the proposed framework.
The empirical work was conducted in the University of Cadiz (Spain), involving a final year course on Operating Systems Administration within a Computer Science degree course on which 43 students were enrolled. The course was coordinated by one of the AMW project members and an author of this article. Learning activities within the course included the development of a wiki-based project, the assessment of which was carried out following the approach proposed here.
In order to develop an authentic assessment experience, the assignment tasks consisted of planning and managing the actual migration process of an enterprise information system. Firstly, the original system was required to consider legacy issues of a system that had been running for certain time, a common task in sysadmin professional role (Brodie & Stonebraker, 1995). Secondly, students were required to use some virtualization (Pearce, Zeadally, & Hunt, 2013) or cloud solution (Bhopale, 2013), two widely demanded technological solution in today’s Information Technology world. Students were divided into thirteen groups of three members and two groups of two members. Each group was required to write its project documentation in a wiki page (the wiki assignment).
Besides, students knew in advance assessment criteria (described in detail in Qualitative assessment workflow section). In particular, two of part of their marks, “A. Team work skills” and “B. Communication and knowledge management” were share by all team members. This moved them toward common goals as a team, and dissuaded students from dividing the project into independent task and work separately. Then, planning and time management was implemented in the mark “D. Final deliverable product”. Students had a deadline to write the assignment. If they finished later, their mark was capped.
The experience carried out during the course involved three stages, which will be described in the approach subsection:
Development of the wiki assignment: This stage began with a seminar, in which students were instructed on how to work collaboratively on a MediaWiki wiki. Students were responsible for planning and managing their assignment, coordinating its tasks and working collaboratively. The wiki was publicly available, and over the six weeks of the project, students made more than 1400 wiki contributions.
Peer- and Self-Assessment of wiki contributions: This stage started with a seminar to teach students how to peer-assess wiki contributions using AMW. Following this, students made 412 qualitative assessments of wiki contributions. This process provided students with critical feedback about their work. Students were also able to respond to an assessment if they disagreed with the mark received.
Teacher refereeing: The teacher conducted a mixed quantitative/qualitative assessment over multiple iterations. The quantitative assessment was made using the information provided by SMW, while the qualitative assessment was supported by AMW.
The research question of this work is the following: “Does a methodology based on the peer- and self-assessment of students’ wiki contributions provide a scalable authentic assessment of their collaborative skills?” From this research question derives the main goals of this work: firstly to provide teachers with a fine-grained assessment of students’ collaborative work on a wiki by engaging students in an authentic assessment experience, and secondly, to achieve the first goal without losing sight of the non-functional requirements of scalability for the entire process.
These goals motivate the design of the methodological framework and also guide the development of the software artefact (the AMW tool) that supports it. This section is divided into two subsections: firstly, the proposed workflow is presented, and secondly, the AMW tool is described.
Qualitative assessment workflow
In this subsection, the methodology for the qualitative assessment of students’ wiki contributions within the context of a wiki assignment is detailed. It consists of three consecutive stages (Fig. 2). During the first stage, each group of students are required to develop their wiki assignment via the collaborative creation of content on the wiki page of their project. The second stage involves the assessment of wiki contributions through a self- and peer-evaluation process. Finally, the third stage is for the refereeing of the teacher, in which he/she performs the following activities: assessment of the final wiki page, resolution of peer assessment replied, review of other peer-assessment not replied and assessment of wiki contributions. These stages are described below.
Stage 1: Development of the wiki assignment
Throughout this stage, each group of students are required to develop their work within the wiki. A series of wiki contributions to an initially empty wiki page is represented in Fig. 3. The author of each wiki contribution is indicated below the arrow representing his/her wiki contribution (between the previous version and the new wiki page). In Fig. 3, R1 represents the first version of a wiki page. Then, a first student makes a contribution, which generates a second version of the wiki page (R2). Later, a second user makes another contribution resulting in a new version (R3). This stage ends when the deadline for the wiki assignment is reached. At this point, the wiki pages are ready to be assessed (the final version of the wiki page, labelled Rf). Although a group of students is responsible for the wiki page (the group of students below the last version), other students on the course may contribute to the wiki page. In this case, the group members decide whether the wiki contribution deserves to be kept, modified, removed or even reported to the teacher (if it is intentionally wrong) in subsequent wiki contributions.
Stage 2: Peer- and self-assessment of wiki contributions
This stage comprises the following activities:
Peer and self-assessment: students conduct peer and self-assessments of wiki contributions using a rubric defined by the teacher. The rubric had several criteria. Each assessment comprises a mark and a comment for each criteria in the rubric. From now on, the descriptor for this assessment will be peer-assessment, as peer and self-assessments are processed in the same way. Each peer-assessment refers only to a wiki contribution made by a single participant, and can therefore be used as a reliable indicator of the actual individual student contribution to the wiki. The student to which the peer-assessment refers is represented in the upper left-hand corner of the assessment report (i.e., the filled rubric) in Fig. 4.
For example, in the peer-assessment illustrated in Fig. 4, the student represented by the striped figure receives the task of assessing a wiki contribution (1). The student checks the resulting wiki page, an overview of the changes between the current revision and the previous one (2). Using this information, the student assesses the wiki contribution filling the rubric, resulting in an assessment report including detailed feedback, i.e., the peer-assessment (3).
Checking assessments: students can check the peer-assessments received, and can see not only the marks received with their comments, but also the link to the wiki contributions to which these refer. In this way, the evidence from the peer-assessment provides formative feedback to the student. The peer-assessment in Fig. 4 shows how the assessment report is available to the assessed student, providing evidence of the assessment (4). The assessor’s identity is anonymized.
Replying: students may reply to any peer-assessment received with which they disagree. Using the same rubric, they must explain the reason for their disagreement. In Fig. 4, the assessed student considers that the peer-assessment is unfair and reports it to the teacher (5). The teacher receives this report notification in the subsequent stage, and referees it.
An interesting issue within the methodology is the question of which wiki contributions are assigned to be assessed by each student. The methodology needs a selection function that chooses relevant contributions to be assessed. The relevance of the assessment of each wiki contribution may vary. For example, wiki contributions affecting a large amount of text may be more interesting than shorter ones. In addition, shorter wiki contributions which only add or remove negative terms may be interesting, since they change the sense of a phrase or paragraph. Even other actions, such as the inclusion of images, may be evidence of the interest of the wiki contribution. Students may also recommend their own wiki contributions as interesting for assessment if they wish, even if they are discarded by the selection function.
Stage 3: Teacher refereeing
This stage comprises three activities: assessment of the final wiki version, resolution of the replies received and, if desired, a review of other assessments not replied.
Assessment of the final wiki page: the teacher assesses the final version of the wiki pages developed by each group of students. This global assessment is necessary since the actual aim of the task is to produce a good final document for the wiki page. As in any other assignment, it must be assessed by the teacher according to the course syllabus. Furthermore, certain assessment criteria can only be evaluated in the final version of the page, such as the coherence of the text. From now on, the descriptor for this assessment will be final-assessment.
Resolution of peer-assessments replied: The teacher resolves the replies, indicating whether they are appropriate or not. If alterations are approved, the relevant grades are modified.
Review of other peer-assessments not replied: The teacher may review a certain number of random peer-assessments even if they were not replied; any marks considered to be wrong are corrected.
Assessment of wiki contributions: If required, the teacher can assess any other wiki contribution.
The information on the reply resolution and other assessments reviewed by the teacher is available anonymously to the students involved (both to the assessors and those assessed).
The qualitative assessment workflow introduced is technologically supported by AMW (anonymized reference). AMW is an open-source web application that, when connected to a MediaWiki installation, enables hetero-, self- and peer-to-peer assessment procedures, while keeping track of the compiled assessment data. In this way, teachers can obtain reports which support the student assessment process. The main features of the application are:
User roles: AMW includes two different user roles: teacher and student. Students can choose three options: assessment of a wiki contribution using the rubric (peer-assessment), a check of their assessed wiki contributions and a review of the wiki contributions that they have assessed. To facilitate this assessment, AMW provides a link showing the differences that the contribution made to the wiki page. The teacher can define the assessment rubric, indicate the number of peer-assessments each student has to make and check the students’ peer-assessments.
Selection function: AMW implements a partially random selection function. When a student requests a wiki contribution for assessment, this is randomly chosen from the largest 30% of the contributions to the wiki which have not already been assessed.
Review and reply system: when checking peer-assessments, students can review the marks they have received and the feedback provided, and see the particular wiki contribution to which their mark refers. For instance, Fig. 5 shows the formative feedback that students receive for one of their assessed contribution (left screenshot) and the wiki contribution referred to (right screenshot). If a student does not agree with the mark received in a peer-assessment, they can respond. In this case, they are provided with the same rubric to indicate the criteria with which they disagree, add the mark they believe they deserve and explain the reason for this in a description field. Later, the teacher must check each case and decide whether or not to approve it. Both the peer-assessment and the reply form are anonymous for the students, although not for the teacher.
This subsection evaluates the achievement of the objectives posed in the previous subsection. Firstly, a detailed explanation is provided for how each skill is assessed in the authentic assessment method. Following this, the improvement in scalability is evaluated.
Scalability assessment experiment
The proposal for the skill assessment is detailed below and is summarized in Table 1. This proposal is based on the course syllabus. Depending on the specific wiki assignments and experiment settings, a teacher may use these indicators as proposed, adapt them for grading other skills, or define new ones.
A. Teamwork skill
This measures the ability of students to work collaboratively.
First iteration: teamwork skill were measured by examining the ratio of students who had contributed several times to the same wiki page in their project. Using SMW, the teacher can see whether the students have worked together by checking that all of them have contributed to the same wiki page. This criterion was based on a coarse-grained indicator, and was relatively easy for all students to achieve, even if they did not work as a team.
Second iteration: the teacher could also detect whether the students had actually collaborated, because they had contributed to the same criterion of the project. This dimension measures the criteria contributed by each user to the wiki page. The teacher considered that a student in a team contributed to a technical criterion (Cr1 to Cr10) of a project if that criterion is assessed on the rubric of any of their assessed wiki contributions (through the peer-assessments received). In Table 2, the criteria assessed for each member of Project13 are shown. While User1 had five rated criteria, User2 and User3 were rated only on one. The only criterion worked on by more than one member (User1 and User2) was Cr8. Thus, in the teacher’s interpretation, these two students worked collaboratively, while User3 did not.
B. Communication and knowledge application skill
This measures the ability to apply knowledge within a practical situation and the ability to communicate with colleagues within the development of a project.
First iteration: the teacher assessed this skill according to the number of team members who contributed at least 20% to the final wiki page version byte count (for groups of three members). In Fig. 6 Work distribution chart of students of Project4 obtained via SMW, a work distribution chart is displayed, showing the ratio of total bytes contributed by each student of Project4. User13 (dark area) contributed 41.8%, User15 (light area) 27.8% and User14 (striped area) 30.4%. Since they all contributed more than the threshold, the teacher considered that they worked collaboratively to develop the project.
Second iteration this skill was assessed as the average of the marks received by students in a team. This indicator assesses the proficiency of the teamwork contributing to the project’s success. Average marks in this contribution may indicate poor project contributions or that a certain wiki contribution obtained good marks for some criteria and an average mark for others, indicating deficient communication between the team members or a limited commitment to the global aim. The mark is calculated using the formula SRAG/NRAG, where SRAG is the sum of the marks the students in a team received through the peer-assessment of their wiki contributions, and NRAG is the number of marks they received. The mark of Project4 (GRDG) is shown in Table 3.
It should be noted that some of User15’s wiki contributions consisted in moving long pieces of text within the wiki page. Considering its quantitative value, SMW adds this to the student’s statistics, providing a limited assessment indicator. Although a detailed review may show that this provided a limited value to the project, a review of each wiki contribution would not be scalable. It can therefore be concluded that under the qualitative approach, this type of wiki contribution can be easily detected.
C. Individual and critical skills
This measures the ability to produce and maintain the quality of the project.
First iteration: the byte contribution timeline profile was measured. For example, the wiki contributions of User15 and User14 are shown respectively in Fig. 7. While both are stepwise profiles, they are not the same: User15 made all of these wiki contributions within just 10 days, while User14 worked for three weeks.
Second iteration: the mark in this dimension (GRDs) is the average of the marks received for each student (through the peer-assessments), expressed by the formula SRAs / NRAs, where SRAs is the sum of marks received by each student and NRAs is the number of marks received by each student. The students’ marks for Project4 are shown in Table 3. Marks for members of Project4, showing a difference of three points out of 10 between User14 and User15, meaning that User15 contributed more to the quality of the project than User14. Again, the qualitative approach provides a more detailed indicator than the quantitative one.
Third iteration: in this case, a third iteration was deployed taking into account the replies that each student’s peer-assessment received in order to assess his/her critical thinking skills. As part of the instruction process, students received clear instructions on the peer-assessment task. Thus, their performance can be used as evidence for the skill of critically assessing their colleagues’ work. Each student started with 10 points in this dimension, and lost 2.5 points for each peer-assessment they made that was corrected by the teacher.
D. Final deliverable product
The final result of a project was assessed as for an enterprise project, where the result must meet stakeholders’ requirements. In this way, all wiki contributions which are not assessed will also be implicitly considered as a whole.
In this approach, the wiki assignment of each group had its final-assessment following the rubric defined by the teacher (Table 4). It has a final-mark which ranged between 0 and 10 that was calculated summing the criteria of the rubric. These criteria were assessed by the teacher once the deadline had been reached. The final-mark was the same for all the students in each team, as all of them (as a team) were responsible for the final result of the project.
One of the objectives of this experience was to perform a qualitative assessment of the students’ work on the wiki by assessing their wiki contributions. In previous experiments, the teacher did not consider the qualitative assessment of wiki contributions since this would take too long. From a theoretical point of view, it can be considered that the amount of time required to assess a wiki contribution i T = t_page. Thus, the time required to assess a number n of wiki contributions is T = n* t_page.
As mentioned above, students made more than 1400 wiki contributions within their 15 wiki pages. The teacher assessed the final version of each page, i.e., the theoretical time required to assess these was T = 15*t_page. Due to the peer and self-assessment stage, the teacher received 412 extra qualitative assessments of wiki contributions. Thus, the teacher had 427 qualitative assessments (412 performed by students and 15 by the teacher); however, the time required for these was the time required by the teacher to perform the assessments of the 15 final version of the wiki pages (T = 15*t_page).
Therefore, the time required by a teacher to assess 427 wiki contributions is T = 427*t_page. This is more than 3000% of the time required following the qualitative approach presented in this methodological framework.