1 Introduction

As the use of social software applications has spread among a substantial part of the world’s population, the educational community has sensed multiple pedagogical potentialities in them, which it has tried to exploit to improve learning processes. These platforms offer a series of properties, such as the possibility of easily including a wide range of content (text, image, audio, video, etc.), of interacting with other users both inside and outside the academic environment, or great flexibility to do everything from anywhere and at any time. A priori, these properties make social software applications an attractive environment in which to build more or less informal and horizontal learning initiatives based on multisensory experimentation (Virtanen et al., 2013), the collective and connective management of information (Goldie, 2016) and the cultivation of processes of social construction of knowledge, all resulting in more reflective, meaningful and socially situated learning (Cochrane & Bateman, 2010; Desselle, 2017).

One of the most popular social software applications is the micro-messaging platform Twitter (Tang & Hew, 2017). It is a free, online and multiplatform service that has been active since 2006. It offers its 200 million daily active users the possibility to post micro-messages (tweets) of up to 280 characters (140, until 2017), including links and multimedia elements, which are openly shared with everyone (Twitter, Inc., 2021). However, the design of the software promotes the formation of networked virtual communities through options such as following (follow) and interacting with certain personal or institutional accounts and creating thematic forums through the classification of messages with the use of hashtags (#).

Contributions addressing the educational use of Twitter have identified teacher and student engagement with the practical experience as one of the key elements that explain the generation, or lack thereof, of learning (Gao et al., 2012; Junco et al., 2013; Malik et al., 2019; Ross et al., 2015; Tang & Hew, 2017). However, as emphasised by authors such as Tess (2013), Lackovic et al. (2017), Williams (2020) and Nkomo et al. (2021), there is still a lack of independent and unbiased research on how engagement and learning are potentially related, as well as a more critical reflection on the appropriate methodologies to assess this relationship.

In this article, these two concepts, engagement and learning, as well as the relationship between them, are studied on the basis of an initiative of continuous educational use of Twitter in three groups of first-year students of the Political Science degree at the University of Valencia (Spain). The activity was designed with the aim of generating a habit of consulting and exchanging current information, published in reputable media, and that this could be discussed reflectively in a collective way. In addition to enjoying great popularity in Spain (7.5 million users or 15% of the population according to We are social and Hootsuite, 2021), Twitter seemed to be the ideal platform for this task compared to other alternatives, given that the system itself invites its users to answer the question: “What’s happening?” and there are precedents of similar applications (Halpin, 2016; Jacquemin et al., 2014; Lowe & Laffey, 2011; Middleditch et al., 2022). Twitter has been proven to be a useful platform for the exchange of information between students (Hosen et al., 2021). It has also been recognized as a virtual agora where cognitive reflection influences the selection and sharing of current affairs information, making it a suitable laboratory to show and promote classroom discussions on issues relevant to the formation of skills in degrees offered in the social sciences (Mosleh et al., 2021). As detailed in the methodological section, the experience was structured in several sub-activities: one of them gave students the role of information receiver, the second placed them as information searchers and transmitters, and the third incorporated a collective participation aspect. The questions guiding this research are:

  • Research question 1: Can the use of Twitter generate student engagement and participation in a practical activity aimed at receiving, exchanging, and discussing the news?

  • Research question 2: Can the use of Twitter during the activity have a positive impact on students’ learning process?

  • Research question 3: Does a relationship exist between the different levels of engagement recorded among students and their academic results?

The article is structured as follows. After this introduction, a review of the scientific literature is synthetically presented, which helps to define and characterise the concepts of engagement and learning. The following methodological section presents the practical activity implemented in the study groups and describes the methods used to analyse the information. The results section presents the quantitative and qualitative outputs. In the discussion section, a critical approach to the implications of these results for teaching and learning as well as for research in evaluation methodologies is made. The conclusions section closes the work.

2 Theoretical background

As fundamental pillars of this research, it is necessary to delve deeper into the concepts of engagement and learning, and to show, through a review of the scientific literature, what evidence exists on the impact of the use of Twitter on them.

2.1 Engagement

While there is still no single, widely accepted definition of the meaning of engagement among the scientific community (Nkomo et al., 2021), Kuh (2009, p. 683) states that “Student engagement represents the time and effort students devote to activities that are empirically linked to desired outcomes of college and what institutions do to induce students to participate in these activities”. By applying this definition to the use of tools such as Twitter in higher education, greater engagement would translate into greater student participation in both curricular and extra-curricular activities organised within the university community (Junco et al., 2013). For some authors, engagement is directly related to the notions of motivation, commitment, expectation and attitude towards the learning activities developed in the subjects (Fraj-Andrés et al., 2022; Hitchcock & Young, 2016; Rinaldo et al., 2011; Watkins, 2017).

A greater amount of evidence reported in the literature than can be provided here indicates that the use of Twitter for uses related to the transmission of information, the organisation of activities and collaborative problem solving in university teaching can contribute to motivating students, increasing their commitment to the progress of the group and making them more involved in the activities and discussions (Erhel et al., 2022; Ricoy & Feliz, 2016; Ross et al., 2015). These benefits or advantages are maximized when the educator shows strong involvement in the task and in providing feedback (Camas et al., 2021; Junco et al., 2013), when the objectives of the online activity have been clearly communicated to the students within a pedagogical project (Fraj-Andrés et al., 2022; Junco et al., 2013; Lackovic et al., 2017), and when this activity is combined with face-to-face discussions about its development in the classroom (Froment et al., 2017; Lowe & Laffey, 2011; Sadowski et al., 2017).

2.2 Learning

Although learning is intrinsic to the development of almost any activity throughout a person’s life, it is patent that higher education focuses its efforts on stimulating certain facets, both through formal teaching methodologies and through practical or applied activities of a more or less informal nature (Kassens-Noor, 2012). The use of Twitter falls into the latter group, and has demonstrated its ability to develop knowledge and skills in a wide range of learning fields. From the outset, its use fosters the formation of a digital culture among learners who, despite their widespread familiarity with the use of digital platforms, have often not considered the best uses or their limits (Gillen & Merchant, 2013; Nagle, 2018).

According to multiple investigations, Twitter can improve students’ communication skills. This would be related to the effort of conciseness required to select information and express ideas and opinions in the limited space of 280 characters (Kassens, 2014). For some authors, Twitter improves problem-solving skills and autonomous learning (Htay et al., 2020; Seifert, 2016). Moreover, when used as a collaborative medium, it also enhances collaborative learning and the construction of formative and professional networks (Malik et al., 2019; Prestridge, 2014; Ricoy & Feliz, 2016). This is evidenced by experimental research showing improved performance in groups exposed to Twitter compared to non-exposed control groups (Junco et al., 2011; Kassens-Noor, 2012; Vooren, 2013). At a deeper level, Twitter is credited with the ability to foster reflective learning, the emergence of metacognitive processes and even its suitability for the cultivation of democratic and ethical values (Evans, 2014; Nagle, 2018; Prestridge, 2014).

3 Methods

To evaluate the degree of engagement and learning resulting from the use of Twitter for teaching, some authors adopt or adapt generic assessment methodologies (Junco et al., 2011; Muñoz-Expósito et al., 2017; Welch & Bonnan-White, 2012). Some authors use quantitative indicators that directly reflect the degree of participation, such as the number of messages contributed or the number of interactions recorded (Gao et al., 2012; Lin et al., 2013; Tiernan, 2014; West et al., 2015). Others rely on teachers’ critical observation of the development of the activity, focusing on the quality of the messages and interactions (Ebner et al., 2010; Junco et al., 2013). Finally, other experiences have made use of subjective assessments made by the students themselves, often standardised through the use of Likert scales, to find out how students have perceived their degree of involvement with the activity (Deaves et al., 2019; Evans, 2014; Fraj-Andrés et al., 2022; Welch & Bonnan-White, 2012). The research combines, through a mixed-methods approach, quantitative indicators inspired by the referenced works, with a qualitative evaluation of both the significance of the contributions and interactions of the learners and their own reflections, presented through a free-content report.

Figures 1 and 2 present an outline of the research design. Figure 1 presents the teaching proposition in which the students participated, including the Twitter activity on which we have focused, as well as other evaluative activities that the students had to complete until they obtained a final mark. Details on implementation are presented in subsection 3.1.

Fig. 1
figure 1

Methodological scheme leading to the extraction of indicators

Full size image
Fig. 2
figure 2

Methodological scheme supporting the assessment of engagement-learning relationships

Full size image

Figure 1 also includes the extraction of indicators of engagement and learning from the actions undertaken by the students and the deliverables of these actions. Figure 2 represents the subsequent study of the relationships between these groups of indicators, approached with a mix of quantitative and qualitative techniques, as shown in the “Engagement-learning assessment” box. This part of the methodological scheme, research-oriented and without direct student participation, is developed in subsection 3.2.

3.1 Implementation of the twitter activity

The activity developed with Twitter would be one of the three practical activities necessary to complete the subject of Human Geography, taught in the degree of Political Science at the University of Valencia (Spain), during the first term of the academic year 2020–2021. Ninety-three students completed the assignment and were considered for this study. The students were divided into three groups. The PA group (42 members) and the PB group (30 members) consisted of first year students, for whom the subject was compulsory. The PS group (21 members) was composed of second-year students of the double degree in Political Science and Sociology. For this group the subject was optional, and all the students had chosen it as their first choice.

The activity was presented as compulsory, with a weight of 5% on the total score of the course. The instructions were shared through the students’ digital portal (Moodle) and were also presented in class. The face-to-face session was used to insist on its pedagogical objective of continuous learning and to clear up doubts. The activity included three sub-tasks with student participation, as well as the submission of a short report summarising the experience and an assessment of it:

  1. A.

    Educator tweets. Each day, the educator would tweet one or several links to news of interest, blog posts or opinion pieces relevant to the subject through an account specifically set up for this purpose (@GeoHumana_UV). The group hashtags “#GH20_PA”, “#GH20_PB” and “#GH20_PS” would always be used to facilitate tracking.

  2. B.

    Learner tweets. Each student had to choose a current topic to follow during the term, always within the scope of the issues dealt with in the syllabus. A minimum number of 5 tweets was compulsory and a maximum of around 10 was suggested.

  3. C.

    Learner-learner and learner-educator discussion. In the participatory dimension, students were encouraged to interact with the educator and their classmates to discuss the content shared through the group’s hashtag. To do so, they could make use of the platform’s tools: like, retweet and comment.

  4. D.

    Reporting and learner self-assessment. Each student was required to write and submit a short report (maximum 1000 words) on the chosen topic and to evaluate the learning experience over the course of the activity.

The final exam for the course afforded students the opportunity to demonstrate what they had learnt by following current affairs through Twitter in at least 8 of the 10 possible points they could obtain, through questions that required reflective skills and connection to current affairs.

3.2 Methods for the analysis of results

The results and outputs of the implementation of the activity were used to extract indicators of engagement, learning and, finally, all were used to study their relationships.

  • Engagement

Using the Twitter Analytics tool on the course profile (https://analytics.twitter.com/user/GeoHumana_UV/home), indicators were extracted for the way in which students interacted with the tweets issued by the lecturer in charge of the subject (Fig. 1). An individualised count of tweets issued by the students was made.

The participatory aspect of the experience enabled the degree of engagement of each student to be measured through a count of the number of replies to tweets and threads with which they had interacted using the hashtag corresponding to their group.

  • Learning

There were three learning indicators (Fig. 1). Firstly, and as an illustration of performance at a very general level, the grade obtained by each student in the subject was used, in a range between 0.0 and 10.0. Secondly, after the final test, it was recorded which students demonstrated the ability to connect the content offered by the teacher in class with the contribution of data, examples, events or evidence derived from the use of Twitter, outside the content provided by the teacher or by other students. Thirdly, a qualitative analysis was made of what the students contributed in their reports as an assessment of the learning experience.

This qualitative analysis was carried out through a thematic coding of the students’ texts, which was developed iteratively with the support of MAXQDA software (Guest et al. 2011). The categorisation system finally crystallised into 4 categories, relating to:

  1. I.

    Content learning: the assessment refers exclusively to the specific study topic chosen by the student.

  2. II.

    Independent learning: the assessment relates to the contribution made by the activity to the student’s learning and skills development.

  3. III.

    Collaborative learning: references are made to aspects of interaction between participants.

  4. IV.

    Reflective learning: where reflection is made on the pedagogical value of the use of Twitter.

  • Assessment of engagement-learning relationships

Quantitative information on individual learner engagement was used to classify students into two categories: Engaged learner and Non-engaged learner. The first group included all students who had shown some form of engagement with the activity, contributing more than 8 tweets and showing any degree of interaction through likes, retweets and comments. The Non-engaged learner group would include the rest of the sample, who only contributed between 5 and 8 tweets and did not show any kind of interaction. This classification was used as a grouping variable in two independent sample t-tests run with SPSS 23 (Fig. 2).

The results of the coding of the students’ subjective assessments of the learning experience were treated qualitatively, as will be presented in the following section.

4 Results

The results of each of the major blocks of analysis described in the methodology and illustrated in Fig. 2 are presented in three subsections.

4.1 Engagement indicators

The 93 students and the teacher who participated in the activity issued, between 16/9/2020 and 15/12/2020, a total of 814 tweets. Following the scheme presented in Fig. 1, the results of the engagement indicators can be summarised as follows.

  1. A.

    Count of educator tweets, impressions, likes and retweets

Online Resource 1 shows the data recorded by Twitter Analytics for the 169 tweets issued by the teacher during the study period. The table includes data such as the id of each tweet, its permalink, its content, the date and time it was issued, as well as a series of counts on the impact and degree of interaction generated. Table 1 is a quantitative summary of this data

Table 1 Summary of impressions and engagements with tweets posted by the teacher.
Full size table

Across the 91-day period studied, the total number of impressions was close to 68,000. However, 3 of the 169 tweets went viral and accumulated, solely among them, a total of more than 23,000 views (Fig. 3). The median number of impressions per tweet, which is much more significant than the total or average in this case, was 275. Bearing in mind that this figure is higher than the number of people enrolled and that, with the exception of 6 students, all of them were followers of the lecturer’s account, it can be assumed that the students regularly received the messages issued by the teacher and read, at least, the headline of the posts.

Fig. 3
figure 3

Tweet activity on the lecturer’s account. Source: Twitter Analytics, 8/21/2021

Full size image

However, the figures in Table 1 also show that the degree of student engagement with the teacher’s tweets was very shallow. On only 281 occasions did someone follow the link to read the full shared news item. There is no way of knowing how many of these clicks were made by students, nor what percentage of those who clicked read the content in full. At best, assuming that all clicks were made by students in the subject and that all read the linked content in full, the actual reading rate over the potential (93 students reading 169 tweets would result in 15,717 expected clicks) would be 1.79%. This is a gross overestimate, especially knowing that most of the impressions were made by people (and probably some bots) outside the subject group.

The statistics pertaining to likes and retweets show a very similar pattern, indicative of low engagement. For these two statistics it was possible to identify the origin of the interactions, with only 5 likes attributable to 3 students and 1 retweet from the same source. The remaining interactions (46 likes and 41 retweets) were produced by journalists, public institutions, non-profit organisations, politicians, and individual users with no apparent link to the subject. These individuals and groups, some with many followers, were responsible for the viralisation of some tweets.

  1. B.

    Count of learner tweets

Online Resource 2 collects, anonymously and classified by groups, data related to the activity and performance of the students involved in the exercise. Blank boxes indicate fields that are unknown, often because the student proceeded to delete his/her Twitter account after completing the course and receiving their grade.

A quantitative synthesis of the data indicates that the 93 participants issued a total of 645 tweets, with a mean of 7 and a median of 6. Only 22 students showed a high level of engagement, issuing 9 tweets or more. Of these, 13 belonged to the small PS group of 21 members. As Table 2 shows, the impact of the students’ tweets in the form of likes and retweets was low. The vast majority of these interactions were with people outside the university group, as some of the participating students used their personal accounts, which together accumulate more than 5000 followers.

Table 2 Summary of learner tweets, likes and retweets
Full size table
  1. III.

    Count of learner-learner, learner-educator and other replies

None of the tweets issued by the lecturer received any response from any enrolled student. They did, however, receive 6 responses from people outside the university. This could have been a good opportunity for students to interact and debate with individuals from outside the university, some of them real specialists in the field or holding public responsibilities.

The total number of comments received for the tweets posted by the students was 20, but only 2 of these came from a fellow classmate. Of the rest, 4 were comments from the teacher and the remaining portion were from people outside the group. None of the authors of the original tweets pursued the conversation with the teacher or with the people who reacted to their messages.

4.2 Learning indicators

The overall performance indicator for each student’s final grade in the course indicates that the 89 people who completed the course and sat the exam obtained an average grade of 6.0 out of 10, with a standard deviation of 2.25 points. It is worth noting that, while the mean for members of the PA and PB groups is only slightly above the pass mark (5.3 and 5.2 respectively), the mean mark in the PS group is significantly higher, at 8.4.

The evidence of learning through the Twitter activity found in the final tests was scarce, with only 4 observations from 3 students, all of them belonging to the PS group.

The qualitative analysis of the learning experience evaluation reports shows, in the first place, that only 36 of the 93 students (39%) who submitted the memory report devoted a few lines to this requirement. An English translation of these sections is included in Online Resource 2. The translation of the texts provided was carried out automatically using the online translators DeepL and Google Translator. Based on this sample, and after applying the coding and analysis system described in section 3, some observations can be made about the subjective perception of learning.

Thirteen students mentioned the acquisition of the habit of consulting information and reflecting on current affairs in the world (Cat. I). In all cases, the adoption of this routine is presented as a valuable practice for the development of learning and is valued very positively. These findings would indicate not only the acquisition of independent learning skills developed through the activity (Cat. II), but also a certain capacity for reflective analysis on the part of approximately one third of the students (Cat. IV).

Only 6 students referred to the collaborative aspect of the activity (Cat. III). Four of them (#12, #29, #71, #72) stressed how enriching it was to receive tweets from classmates and share ideas with them. These findings point to a social use of Twitter as a tool for generating group cohesion. A fifth student (#80), on the other hand, reflects on how the activity, carried out through her personal Twitter account, allowed her to make her followers aware of the injustice of the phenomenon known as “voluntourism”. This social use of the activity was not highlighted by any other participant, nor was there any mention of actors outside the activity. One student (#84) did draw attention to the need to find ways to encourage interaction during the activity.

The reflective learning aspect (Cat. IV) shows that, in general, the activity was very well received by the students, most of whom emphasised their satisfaction at using an unconventional learning method and, for some, discovered a more serious, academic, or professional use of Twitter, complementary to its social or recreational use (#11, #26, #29, #49, #57, #58, #59, #67, #69, #84, #86). Several students underlined how the diversity of voices and opinions exchanged through the task allowed them to see the multiple perspectives from which changes in the world can be observed, enabling them to be more critical of the way in which these transformations are presented in the media (#12, #25, #29, #38, #39, #46, #57, #58, #76, #85, #93). Some of the evaluations suggested that the activity was considered to be well aligned with the expectations that the Political Science degree generates among the students and that it helped some of them to become more involved in these academic studies (#25, #39, #60). Two students expressed constructive criticism of some aspects of the activity, for its rigidity and lack of value added to the subject (#19) and for the limited length allowed for the writing of the report (#52).

4.3 Engagement-learning relationships

The clustering exercise that preceded the analysis of the relationship between engagement and learning yielded a classification of 27 students in the Engaged category (29%) and 66 (71%) in the Non-engaged group (Online Resource 2). The distribution by class-groups for the Engaged students is as follows: 5/42 (11.9%) in the PA group, 8/26 (30.8%) in the PB group and 14/21 (66.7%) in the PS group.

The mean final grade for students in the Engaged group was 7.1/10, with a standard deviation of 2.19, while the mean final grade for students classified as Non-engaged was 5.54, with a standard deviation of 2.12. The independent samples t-test reveals how the differences in the means between the two groups are statistically significant, both assuming equal variance (t(87) = 3.156, p = .002) and not (t(48.104) = 3.115, p = .003). If the same test is performed only for the 68 components of the PA and PB groups with final grades, more similar in their assessment results compared to the performance of the PS group, the difference between Engaged (M = 5.6, SD = 2.23) and Non-engaged (M = 5.2, SD = 2.0) individuals disappears completely (t(66) = .579, p = .564; t(16.932) = .542, p = .594), and no differences are identified between the two groups.

The scarcity of cases in which students were identified as having made use of what they had learned through the Twitter activity in the written exam and the fact that all the cases correspond to students classified as Engaged, renders the planned running of an independent samples T-test analysis impossible. Despite the weakness of the sample, the fact that the three students who made fruitful use of the Twitter activity in the exam belonged to the minority group of Engaged students and all belonged to the PS teaching group can be qualitatively appraised.

The qualitative analysis of the relationship between the degree of engagement and self-assessment of learning through the Twitter activity yielded somewhat disconcerting results. Of the 36 students who carried out the self-assessment exercise and rated different aspects of the experience very positively (see section 4.2), 28 of them (77.8%) were part of the Non-engaged group. This contingent had barely issued the minimum of 5 mandatory tweets, in most cases had done so during the last two weeks of the course and had not interacted with any classmates or the teacher. Despite this evident lack of engagement with the activity, these students manifest some of the most enthusiastic, even hyperbolic, evaluations of learning (see, for example, cases #9, #12, #26, #28, #36, #57, #59, #65 and #72 in Online Resource 2). Two of these minimally participative students (#12 and #72) went so far as to praise the collaborative aspect of the learning achieved.

5 Discussion

This discussion will address the task of answering each of the three research questions and, based on the results obtained, critically assess some methodological implications.

5.1 The production of engagement

Some informal observations at the beginning of the experience suggested that the level of student engagement would be stimulated by the type of activity proposed: when it was presented in class it was received with a certain amount of enthusiasm, a group of students from a previous year expressed envy towards their classmates for not having been able to use Twitter when they took the subject, and a student from a different degree, on finding out about the activity through a friend who was taking a Political Science BA, asked the teacher if it could also be implemented in his group. According to authors such as Fraj-Andrés et al. (2022) or Chatterjee and Parra (2022), the existence or generation of high levels of motivation or expectation such as the above is a key ingredient for the success of teaching experiences with this type of educational tool. Adams et al. (2018) observed similarly promising starts in their experiences with Twitter for teaching. However, our expectations turned out, in the light of the results, to be completely unfounded.

The indicators of quantity and quality of participation and interaction presented in section 4.1 show a very low degree of student engagement with the activity. Despite receiving the teacher’s links, only 1.79%, at most, followed them. The majority of students only shared a minimal amount of content, typically on the final days of an activity that was presented as continuous. Hardly anyone took the opportunity to interact with the teacher, peers, or the professionals and public figures who did respond to some of the messages, making them viral. These missed opportunities lead to a negative answer to research question #1: the mere use of Twitter does not, in itself, lead to engagement and participation of students in an information exchange and discussion activity (see also Adams et al., 2018; Rinaldo et al., 2011; Seale et al., 2015; Sun et al., 2014; Welch & Bonnan-White, 2012). Our finding is not incompatible with the fact that an illusion of engagement is generated, both by the teacher and the students, as shown by the anecdotal observations that opened this section and by many of the self-assessment reports of the students, an issue that is discussed later.

Although the degree of engagement was shown to be very low, it is also true that the results illustrate a relatively greater involvement on the part of the students in the PS group, made up of students who had chosen the subject, which was optional for them, as their first option. This could indicate that the degree of initial motivation of the students would be a very important factor in determining engagement with the activities, regardless of their nature. After an experience with Twitter, a similar observation was reported by Evans (2014), and it underscores how a reportedly successful use of Twitter to generate engagement by learners in higher education settings may simply be detecting a certain degree of engagement, often catalysed by other factors. To assess the impact of the technology on this variable, experimental designs that control for the effect of these external influences are indispensable.

In terms of engagement metrics, the research has revealed two other subtleties that should be taken into account when assessing this dimension. One relates to the record of usage and interaction with the dedicated hashtags. As has been pointed out, and even though no active effort was made to project the activity outside the teaching groups, most of the interactions with the teacher and student tweets with each group’s hashtag came from people and institutions outside the university. Recording only the number of likes, retweets and comments received by messages with the hashtag, as some analysis tools allow, would lead, once again, to magnifying the perception of engagement in the activity on the part of the students. It is important to couple the collection of quantitative indicators of engagement with a detailed analysis of the sources of tweet dissemination.

Secondly, it has been found that the number of impressions is also a very poor indicator of the impact of the Twitter activity in terms of generating engagement. Almost 70,000 impressions of the educator’s messages in 91 days and the fact that the 169 tweets received less than 300 clicks in total, indicates that both students and the rest of the user community make a very superficial use of the platform as a source of information, which is limited, in the best of cases, to reading only a headline. This effect could be compounded by two factors. First, as reported by authors like Kassens-Noor (2012) or Malik et al. (2019), many messages appear superficial and convey very limited information, a limitation that is partly caused by the constraints imposed by the limited message length of 280 characters. Secondly, the tweet feed is usually checked on a mobile phone, while careful reading of linked pieces is more convenient on larger devices. As Lackovic (2017, p. 46) diplomatically reminds us, “Just scrolling down and passing on information can be a form of learning experience, but its depth is questionable”.

5.2 From engagement to learning

Despite being an important ingredient, the existence of student engagement in an activity does not systematically guarantee learning. However, it is true that, without engagement, it seems difficult to achieve any meaningful learning. In the case study, it seems unlikely that the very sporadic reading of the information shared by the educator and the virtually non-existent interaction with peers or outsiders could have led to (i) the generation of independent learning that involves the habit of consulting reliable sources on current events, (ii) the collaborative learning that their discussion could produce or (iii) the reflective learning that would entail the confrontation of ideas and perspectives to build a critical personal discourse around the social implications of what has been learned. The results support this notion, except in very specific and exceptional cases. However, it is worth discussing two results, related to the methodological dimension of learning evaluation.

It would have been very easy for the author of this article to solely rely on the results of statistical tests to argue that there is a significant relationship between greater engagement with Twitter and students’ academic performance, measured both in terms of a higher final grade and the beneficial use of what was learned in the exam by some engaged students. However, a closer reading of the results indicates that, if this relationship exists, it is the result of an inhomogeneous sample in terms of qualitative characteristics of the students. This is evident from the comparative analysis made throughout the research between the members of the PA and PB groups, on the one hand, and the PS group, on the other. Although descriptive statistics and similarity tests indicate that the members of the latter group were comparatively more engaged and, at the same time, the best performers, it seems risky to answer the third research question in the affirmative and attribute a causal relationship between the two pieces of evidence. It should be hypothesised, at a minimum, that the students who were more motivated at a general level with their studies, and with the subject, or who aspire to obtain a better result in their gradebook, will make more of an effort to be active and committed to the activity and that, concurrently, they will devote more attention to the preparation of the other activities and the final test. In a similarly strategic way, the rest of the students would be satisfied with fulfilling the minimum requirements to achieve a pass (see also Bunce et al., 2017). Once again, this suggests that, in part, the profusion of apparently satisfactory results of experiences using Twitter on learning in the scientific literature may be due to confusions between statistical results and causal relationships. Authors such as Evans (2014) and West et al. (2015) make similar observations. The importance of minimising, through correct experimental design, the risk of proliferation of these errors of commission, for example through careful segmentation of the samples, should be emphasised.

The other finding of research value from the experience relates to the validity of self-assessments as a tool and method for assessing learning. The pronounced dissonance between the results of use/interaction and self-assessment of learning recorded in the case study, similar to that found by Deaves et al. (2019), should be a warning for studies that exclusively use subjective methods, such as closed surveys, open surveys, such as in this study, or Likert scales, for the evaluation of the impact of Twitter and similar tools. This observation reinforces similar warnings about the reliability of these techniques issued by authors such as Tess (2013), Gao et al. (2012), Tang and Hew (2017) and Nkomo et al. (2021). The replacement of these techniques with others that are more robust or, at least, their complementary application, may be particularly advisable in groups of university students in the first years of their studies. Authors such as Lackovic et al. (2017) have already addressed this issue of timing, expressing similar opinions. We also agree with Chatterjee and Parra (2022), Fraj-Andrés et al. (2022) and Junco et al. (2013) on the need to transmit to students the pedagogical sense of the practical experience, in this case not only as a possible stimulus for their engagement, but also so that they can make a more well-founded self-learning evaluation in keeping with the educator’s expectations and those of each learner.

5.3 Limitations and future research

Some of the main limitations of this research derive from the decisions to implement the academic activity with Twitter as mandatory, evaluated with 5% of the weight in the grade of the subject, linked to a public account by the teacher and giving students the freedom to choose between registering a new account or using their regular one.

The public nature of the teacher’s account makes it difficult to identify the people who generated the impressions and to determine the proportion of clicks whose authorship can be attributed to the participants. These limitations have constrained the accurate measurement of aspects related to student engagement, as shown in Table 1. However, given the small size of the interactions, this limitation has not hindered the extraction of results and conclusions. On the other hand, this decision sought to facilitate the interaction of the students with people outside their group or even outside the university environment. It was hoped that, as it had happened in previous studies, some of the messages would have some impact beyond the academic domain (Malik et al., 2019; McKenzie, 2014; Rinaldo et al., 2011). This would allow students to engage in dialogue or interact with people involved in politics, public administration or other fields related to their future work environment, expanding their professional and personal learning networks. The magnitude of the potential benefits to the students was considered to outweigh the convenience to the research that its development in a more closed environment would provide.

As the results show, this potential, despite inputs from several public figures with thousands of followers, did not materialize. We can only speculate on the causes of this inhibition on the part of the students. In line with the research by Deaves et al. (2019), Lackovic et al. (2017), Malik et al. (2019) and Hosen et al. (2021), it is suspected that the reputational factor related to the projection of an adequate digital identity played a role, also in the experience described above, as a deterrent on the active participation of students. Interacting with established personalities from the world of politics, journalism and public administration carries the risk and nurtures the fear of disseminating an inappropriate message or one that may be judged as lacking in value. This inhibitor may have been reinforced in the case of students who used their personal account for the exercise. The combination of different modalities in the implementation of the activity is a line of future research that would allow to discriminate the impact of this reputational factor as a driver or inhibitor of participation. The range of modalities should incorporate both open and restricted accounts, an offline mode and, as an intermediate point, to experiment with the use of social platforms that prioritize interaction in an online environment where selective and more actively moderated groups participate, as is the case of Mastodon.

The research would also have benefited from greater control of the effect of the incentives on the degree of student involvement in the activity. An obvious incentive is the mandatory nature of the activity and the fact that it contributes to part of the final grade. It is known that the weight of the latter component can affect student performance, but there are no clear guidelines or recommendations in the literature to determine the elasticity of this relationship between grade weight and engagement. (Ebner et al., 2010; Tang & Hew, 2017; West et al., 2015). Yet, there is a research result that opens a new avenue of inquiry: What were the reasons that led a small number of students to blatantly lie, in their reports, about learning deriving from interactions that, in reality, never existed in their case? It can be hypothesized that, in addition to the reputational dimension among peers and in the eyes of recognized figures in the professional field, there is a factor that would drive predominantly less-engaged students to look after the image they project towards the educator in a somewhat vehement manner. Finding out about this factor and how it interacts with others is worthy of further scrutiny in future research.

6 Conclusion

The reporting of rather underwhelming learning experiences involving Twitter and similar platforms, while uncommon (Evans, 2014; but see Lackovic et al., 2017; Lin et al., 2013; Middleditch et al., 2022; Tess, 2013; Thoms & Eryilmaz, 2015; Welch & Bonnan-White, 2012; West et al., 2015), can also offer valuable lessons for the educational community and for the progress of the research.

In the light of the results discussed, the first issue to be addressed is to what extent it is good or necessary to incorporate tools such as Twitter which, for all their apparent potential and wide acceptance by students, do not prove capable of increasing students’ engagement with their own learning. If their effectiveness must be questioned for the purpose of the activity and for the modality implemented in the case study, their efficiency must also be called into question. Much of the specialised literature insists that to achieve success in initiatives of this nature, the educator must be fully involved in the activity and deploy a multiplicity of roles: technical trainer, animator, moderator, pedagogue, evaluator, co-participant, role model, etc. In the study, this multiplication of the teacher did not translate into satisfactory results, questioning the efficiency of the learning process and raising the question of whether, with the lesser effort involved in sharing press clippings in class, a better result could have been achieved. If, on the other hand, the use of these platforms continues to become commonplace, the experience presented here suggests that it will have to be the faculties or departments who will have to consider the possibility of training or incorporating specialists in the role of community manager, to boost the non-pedagogical aspects of the activities.

The research has allowed us to question two of the most used methods for assessing the impact of Twitter use in teaching. First, it has been shown that the existence of statistical correlations between the degree of student engagement and their academic results will not always reveal a causal relationship, as continues to be accepted uncritically on an all-too-often basis. Research designs should consider the characteristics of the sample and incorporate mechanisms to control for the effect of pre-existing motivation and other externalities. Second, the results of students’ self-assessments of learning may be overestimating, by a very wide margin, the actual learning in many of its dimensions. These findings invite researchers to combine quantitative and qualitative assessment methods, generating redundancies that will minimise the risks of undermining or, most often, magnifying the benefits of using Twitter for educational purposes.