Reporting CCI and Learning Technology Research

Abstract

Designing and conducting high quality research is extremely important in CCI and learning technology research. However, the same high-quality needs to be followed during the reporting of the work. At the end of the day, this what the reviewers and readers will credit – therefore, reporting is of equal importance. This chapter provides information on how you should structure your article and the information that is usually required. This chapter is intended to serve as a template for CCI and learning technology researchers. Moreover, I have summarized some recommendations based on my experience, as well as on published guidelines and recommendations from neighboring fields.

For CCI and learning technology research to be disseminated, it is important that it be efficiently reported (and published) in various proceedings, and journals. Reporting CCI and learning technology research has many similarities with other human-factors IT-related fields, such as game technology, computing education research, HCI/interaction design, and software engineering. This is due to the fact that most of these fields rely heavily on the guidelines of the American Psychological Association (APA) for reporting. The recommendations of this section are based on the authors’ experience, as well as on published guidelines and recommendations from the relevant fields (e.g., Ko et al., 2015; Recker, 2012; Wobbrock, 2015; Ross & Morrison, 2013) and the APA journal article reporting standards (JARS) for reporting quantitative research (reporting the findings of a study using numeric representations),¹ qualitative research (reporting the findings of a study using narrative representations),² and mixed methods research (reporting the findings of a study using both narrative and numeric representations).³ Therefore, independently of the statistical analysis software or data science programming language you use, your reporting must follow a standard structure and include the information indicated in the conventions. Several resources are available, including the JARS website (https://apastyle.apa.org/jars), which includes details about how to report your results for a wide range of research designs and statistical analyses. There is also a summary of how to report the results of your statistical analyses provided by Dr. Jeffrey Kahn from Illinois State University (https://psychology.illinoisstate.edu/selandau/ReportingStatisticsinAPAStyle.html). This section provides a brief overview of how to report CCI and learning technology research. Figure 7.1 shows the high-level structure of a typical paper/report.

Fig. 7.1

Typical structure for a learning technology or CCI paper/report

FormalPara Abstract

An abstract is mandatory in most reports/papers. Its purpose is to provide potential readers with an overview of the paper so that they can determine if it is relevant to their research. The abstract should be no longer than 250 words and should briefly cover each of the necessary elements (in one or two sentences per element). Effective abstracts describe three things: (1) what was done, (2) how it was done, and (3) what was found (the main results). Be specific; for instance, instead of “many” or “most,” say “84%.” The abstract should include the following elements. It is generally up to the writer whether to keep the words (here, in bold) that indicate the section labels.

• Contribution. Briefly describe the new knowledge emerging from the study.

• Background. Briefly describe the rationale for the study presented in the manuscript. The background is expected to provide a rationale for the study (i.e., why the study is needed). It is expected to establish a context that suggests the study has broad application in many programs across the world.

• Research questions. Briefly present the RQs that the study addresses.

• Methodology. Briefly describe the research methodology used to conduct the study. This section should also briefly mention limitations of the study (e.g., small sample size).

• Findings. Briefly summarize the findings of the study.

7.1 Introduction (and Motivation)

The introduction is used to explain the motivation of your paper to the reader. Its primary purpose is to explain why you did the work you did. In particular, the introduction needs to address five major points. Although you are also free to structure your introduction in the following way (or not), make sure you cover the five points.

• The topic. This is a description of issues in whatever “world” is relevant to your topic (e.g., learning visualizations, AR/VR technology in education, or game-based learning in the classroom). This part of the introduction begins with general information about the topic and then narrows to the specific focus of the new study (normally in the last sentence of the paragraph). Definitions of key terms may also be provided. In this section, the researcher carefully selects previously published articles to establish a foundation for the study that is being reported. This helps the reader to contextualize the study’s topic(s) and findings. Drawing on the popular press can be helpful in supporting your cause.

• The gap. This is a description of the gap between what is known and what needs to be known. Here, challenges or problems can be framed as an opportunity, as you can motivate your work by connecting it to things that matter to people. “Absence from the literature” alone is not a good justification, although it is useful to add after you have established a problem or opportunity that is worth pursuing in its own right. As a stand-alone motivational statement, however, absence from the literature is not convincing (e.g., maybe the literature is silent on an issue because the issue is not important). Near the end of this section, you should establish a gap in the previously reported studies and identify what questions still need to be answered (e.g., by mentioning the major papers that deal with the topic, what they have accomplished, and their major flaws/omissions/neglected issues). This is an important part of the published study, because it tells the reader how your study’s findings relate to and increase the knowledge base.

• The goal. After describing the gap, you need to describe the goal of the study. Devoting a paragraph to the goal in the introduction is reasonable, as it allows you to describe the research or inquiry question(s) of the study. Sometimes the inquiry question or statement will be in its own section, and sometimes it will form part of the introduction. Either way, this paragraph of the introduction should provide some description of the goal and the respective research objectives/RQs. It should clarify how the proposed research objectives/RQs address the previously identified flaws/omissions/neglected issues and why it is important to address them (not every omission is important, and some issues may not have been addressed because they are trivial).

• What you did and what you found. After clarifying the goal, you need to describe what the study does (in more detail than in the abstract) and offer some key results. It is important to describe what the study does through the lens of the aforementioned goal (showing that you have adopted the optimal approach for addressing your goal). In addition, it is important to give a brief overview of the major findings.

• How this study contributes. A good way to end your introduction is by framing the contributions that your work makes. The contribution can be structured as a bulleted or numbered list within a paragraph. Most papers will make two to three contributions. Overstating your contribution can lead to criticism from reviewers, and even to the rejection of the paper; therefore, whatever expectations and research claims you make in the introduction must be delivered in the rest of the paper and discussed thoroughly in the discussion section. Therefore, it is important to be as clear as possible.

By structuring your introduction around these five major points (one paragraph per point; a maximum of two paragraphs if a particular point needs further elaboration), you succinctly motivate the work in relation to a real problem or opportunity, and you hook the reader with the key results and contributions. Reviewers’ opinions about the importance and quality of a work are formed while reading the introduction. If it reads poorly or is missing key aspects (i.e., if it fails several of the tests implicit in the structure above, such as importance of the topic, a clear gap, concrete research objectives, and specific contributions), reviewers’ opinions can very quickly tilt negatively. Examples of introduction and motivation sections that follow this clean structure can be found here, one in the context of avatars in motion-based educational games (Lee-Cultura et al., 2020), and one in the context of wearable devices for estimating the learning experience (Giannakos et al., 2020).

7.2 Background and Related Work

The primary function of the background and related work section is to answer the following four questions: What are the major/important relevant works in this topical area? What did they do? What did they find? How is this work here different (e.g., in terms of extending or complementing previous work)? This section may also include relevant theories and be called “Related work and background theories” (or similar). The last item you need to consider when writing this section is the differentiation of your work from related studies. This is very important, since it allows the reader to see the big picture of your study and how it furthers knowledge in this particular field.

The background and related work section should not read like a list of who did what. It should offer insights into and education about prior work, portraying recent developments in the field. It should help readers to understand previous work better than they did before. It is advisable to break this section into subsections, as this will enable the reader to grasp the various themes. For instance, if the paper introduces an adaptive game-based learning system for teaching mathematics, it is reasonable to report related work on (1) adaptive learning systems, (2) game-based learning, and (3) technology-assisted teaching of mathematics. The differentiation of the current work from prior work can then be achieved theme by theme, rather than by contrasting it with every piece of prior work under discussion.

The craft of writing this section lies in clarifying what has been done and how the study furthers knowledge. Thus, it is important to be able to formulate the differentiation positively, without being defensive in relation to previous work, and to maintain a narrative flow that nurtures readers’ understanding. It is not necessary for the current work to assert itself as “better” than prior work; rather, it should be different than previous studies and should contribute adequately to an important research objective. This can take the form of asking a different question, using a different method or technique, building on different technological affordances, or focusing on different effects of these affordances (e.g., not necessarily to learning itself, but also to students’ motivation and engagement). It is common for the last sentence of each paragraph to summarize the paragraph and provide insights to motivate the next paragraph. At the end of this section, it is important that the reader understands which gaps the study intends to bridge and the importance of doing so (i.e., the implications for research, theory, and practice).

Some venues might require a mini “literature review” section, but this is not required in all cases. You should bear in mind that this is not a literature review paper, and therefore related works that are not very close to the intended contribution may need to be ignored. This section should allow you to zoom in on the particular field. Zooming in too close (to an area with only two or three relevant studies) or not zooming in close enough (trying to describe all the relevant studies) will fail to provide the necessary information and narrative flow for the reader. Examples of related work sections that follow the aforementioned instructions can be seen in Lee-Cultura et al. (2020) and Giannakos et al. (2020).

7.3 Methods

The research methods employed must be described carefully, in great detail, and in line with accepted standards and conventions. As mentioned above, in the field of CCI and learning technology, forms of APA style are commonly used (APA, 2020). Therefore, drawing on our experience and on the guidelines and recommendations for various human-factors IT-related fields (e.g., Ko et al., 2015; Recker, 2012; Wobbrock, 2015; Ross & Morrison, 2013), we propose the following subsections outline that enables adequate description of methodological decisions. You may encounter small variations on the proposed outline, and differences are found between quantitative, qualitative, and mixed methods research studies. For example, qualitative research studies tend to use a wider range of structures than quantitative research studies. Nevertheless, there are certain essential methodological aspects and decisions that need to be covered, regardless of the nature of the research study. Therefore, the following subsections form part of a common structure found in most learning technology and CCI papers.

7.3.1 Participants

In this subsection, the researcher identifies particular characteristics (variables) of the participants in the study that are relevant to the study design. A typical participants subsection provides the number of participants, their genders, their mean age, their age variance, and the selection process and assignment (e.g., random selection or convenience sampling). An important element in this section is the description of the end-users; for instance, you should clarify the specific group of people you focus on (e.g., primary school students, teachers, or university students) and state your inclusion and exclusion criteria. You should also report whether and how participants were compensated. The rule of thumb is that you should give enough information for a similar group of users to be recruited in the future by an expert reader.

7.3.2 Setting/Procedure

This is normally the longest subsection within the method section, and it should describe the process participants went through during the treatment and data collection (e.g., for a lab study, from their arrival to their departure from the lab; for an in-the-wild study, the settings in the school or other environment). The researcher must identify where the data collection took place and provide details of the setting(s), which can be a single setting, various settings, or even some uncontrolled settings (as in experience sampling method studies). This section should detail what tasks the participants performed and in which order. It is important to provide enough detail for the expert reader to be able to replicate the study.

7.3.3 Data Collection

Researchers collect data in different ways. Whether the design is qualitative, quantitative, or mixed research, you need to describe what kind of data you have collected (e.g., log files, questionnaire data, sensor data, interviews, or field notes). Data collection is associated with specific measurements (which are associated with RQs). In some data collection methods (e.g., questionnaire data and some log files), the measurements are predefined; in others (e.g., sensor data), the measurements are post-computed; and in yet others (commonly in qualitative research studies), there are no measurements. In this subsection, you should report the data collection method and, if relevant, the measurements employed in the study.

7.3.4 Research Design

This subsection describes the experimental design and is a common feature of quantitative research studies and mixed methods research studies. A detailed description of experimental designs has been given in Chap. 4 of this volume (Common types of experimental designs). An effective practice for describing experiments with multiple treatments and/or groups is to set out the treatments and groups and their details. It is important to use the correct nomenclature and include all the necessary details. For example, we might state the following: It was a true experiment with an experimental group and a control group. The control group played a quiz (Kahoot!) at the end of each lecture that gave them feedback at the end of the quiz, whereas the experimental group played a quiz (Kahoot!) at the end of each lecture that gave them immediate feedback after each question. This research design investigates the effect of immediate feedback on students (e.g., performance and attitude).

7.3.5 Data Analysis

In quantitative studies, this subsection may be referred to statistical analysis, but you will normally see it referred to as data analysis. In this subsection, you must describe how the data collected were analyzed in order to answer the RQs (and/or test the hypotheses) of your paper. In other words, you should state what process you employed to make sense of the data you collected. If you collected quantitative data, you should describe the systematic process employed to analyze the numeric information collected, including the formal statistical analysis approach you took. For instance, you might state: “To investigate the effect of immediate feedback during the quiz on learning performance and students’ attitudes, an independent samples t-test was applied between the control and experimental groups.” You should also specify how the analyses were conducted (e.g., “The analyses were performed using SPSS 25.0 for Windows”).

If qualitative data were collected, you will need to describe how they were analyzed. For instance, did you employ inductive or deductive coding? How were the themes identified? How did you arrive at your findings/results? How were objectivity and validity ensured? Did you adopt any “standard” processes (e.g., grounded theory)? If so, you need to describe in detail the procedures you followed for open coding, axial coding, selective coding, and theory formation, as well as providing the coding protocol (e.g., as an appendix or in an online repository). Did you use any reliability measurements (e.g., Cohen’s kappa)? If so, what was the outcome? In general, the rule of thumb is that you should give enough detail for the analysis to be replicated by an expert reader if they have your data.

In a mixed methods research study, you should describe the aforementioned processes and add how you have triangulated your data (i.e., how you reached similar conclusions from different data sources and analyses). Triangulation can significantly strengthen the outcomes of your research, because it allows you to test your hypotheses and to explore ideas and experiences in depth.

7.4 Findings (or Results)

This section is used to report the findings (sometimes called the results) of the study. The findings come from the analysis of the data collected (Chap. 5 of this volume). No interpretation of the findings is made in this section. If the data were quantitative, the findings are reported numerically. If the data were qualitative, the findings are reported using narratives (usually quotations). If the data were quantitative and qualitative, both numeric and narrative representations are reported. The findings section speaks for itself: in it, you should report the results of your work in an organized way. The section refrains from discussing the importance of the results and from describing any potential implications; it focuses on reporting the results. Try to use charts, graphs, and tables as appropriate (for example, Fig. 7.2) and in line with the appropriate reporting conventions (https://www.socscistatistics.com/tutorials/test/default.aspx). Although you should refrain from discussing the results at this point, a well-written findings section has an easy-to-read narrative flow that allows the reader to grasp the answers to the RQs.

Fig. 7.2

Example of visualization of results comparing control and experimental groups

For quantitative research studies, the findings section can be divided into subsections that address different dependent variables or RQs. When including the results of statistical tests, you always need to use the appropriate conventions. For example, you can write: “An independent samples t-test was conducted to compare quiz scores for students who received immediate feedback and students’ who received feedback at the end of the quiz” (as a reminder of which statistical test was used). Then, you will need to report the results clearly; for example, “The results show that there was a significant difference between the scores of students who received immediate feedback (M = 54.99, SD = 8.13) and those who received feedback at the end of the quiz (M = 50.12, SD = 10.31) (t (198) = −3.73, p = 0.000).” Fig. 7.3 shows the output received from IBM SPSS software for this particular example and how you need to report it.

Fig. 7.3

Example reporting of independent samples t-test results from SPSS output

Supplementing statistical significance with descriptive statistics and effect sizes is very important, since it allows the reader to grasp the results better. The APA notes that it is “almost always necessary” to include effects sizes in the results section (APA, 2020). The effect size indicates the number of standard deviations by which the means of the experimental group differ from those of the control group. In the context of learning technology and CCI, an effect size (Cohen’s d) of +0.8 indicates an important effect (i.e., a full standard deviation), while effect sizes of +0.2 and + 0.5 indicate small and medium effects, respectively (Cohen, 1988).

For qualitative field studies, there is a higher degree of freedom in reporting the results; however, the findings section is often divided into subsections according to the themes that emerged. The results sections may be quite long, incorporating notes and observations from the researchers as well as direct quotations from the participants (e.g., from the post-interview or from recordings made during the treatment). Organizing your results into subsections makes it easier for the reader to follow the flow of your paper.

Learning technology papers that adhere to the aforementioned conventions in the method and results sections include Ahn et al. (2018) and Hiniker et al. (2018) for qualitative research, Papamitsiou et al. (2019) and Papavlasopoulou et al. (2018) for quantitative research, and Watson et al. (2017) for mixed methods research. As you can see from these examples, although there is some degree of freedom in writing these sections, the specified general structure and content must be present.

7.5 Discussion

In the discussion section, the researcher interprets the results of the study. In the first paragraph, the researcher summarizes the main findings and relates them to the initial problem that the paper set out to address (i.e., the RQs). The summary of the findings and their connection to the RQs should come early in the discussion section. Then, the interpretation should begin. What do the results mean? What are the reasons behind the results? What do the results tell you in light of the relevant theory and/or related works? Each finding needs to be discussed in detail and interpreted against related published works. Does it confirm, disconfirm, or extend their results? This step is essential in demonstrating your research contribution (how your research adds to, complements, or clarifies the current body of knowledge). The goal of this section is to enable the reader to understand what your findings mean. Connections are usually made with the knowledge that was established in the introduction and related work sections, showing how your findings answer the RQs and bridge the knowledge gap you identified in the introduction. This is a very important part of the discussion, since it clarifies the contribution of your paper (i.e., what it adds to previously published works). Limited contribution is one of the most common reasons for a paper being rejected.

In other words, the discussion section reports what the results mean, what is interesting (novel) about them, and why they matter. The third part of the discussion, focused on why the results matter, is called the “implications” of the results. The implications part explains what the results mean, whether and how they influence current research and practice, and whether and how they will have any impact on theory. Although the discussion is not usually divided into subsections, it can be if it is long; in that case, the implications usually form a stand-alone subsection.

Another important part of the discussion that can be a stand-alone subsection is the limitations of the study. These are usually included at the end of the discussion section in the form of a paragraph or two summarizing the limitations caused by your methodological decisions. When selecting your method, there are always tradeoffs between the various decisions (e.g., research design and measurement instruments), and a mature and reflective researcher should state those limitations and view the results through that lens. In this section, it is important to avoid speculating about matters that do not emerge from the data collected. After limited contribution, speculation is the second most common sign of a low-quality discussion section. Good examples of learning technology and CCI papers that follow the abovementioned logic in the discussion section are Lee-Cultura et al. (2020) and Giannakos et al. (2020).

7.6 Conclusions and Further Research

This is a very short section (around two paragraphs) that addresses three points:

• It summarizes the contributions of the work and clarifies that you have delivered what has been promised in the RQs/introduction;

• It highlights any key points that you would like the reader to remember (i.e., the take-home message);

• It emphasizes the specific significance of the work and calls for future research that is likely to extend the reported study’s findings (e.g., showing how this work opens avenues for new research).

Since the contributions have already been reported and discussed in previous sections, it is important to zoom out in this section and try to see the wood rather than the trees. You should avoid copying and pasting the text used in other sections to describe the contribution; however, limited repetition (such as rewriting some text in a simplified manner) may be appropriate. Try to frame the contributions of the work in such a way that their value can be understood by a generalist, not just by researchers of this particular narrow topic.

The last paragraph of this section describes the avenues for future research that have been opened by your results and calls for studies that will extend, complement, or exploit your findings. In some cases, researchers use this part to describe their own future research. It is often better to suggest a few well-considered and important future steps than a “mainstream” list of smaller items (e.g., collecting additional data or implementing a similar study).