1 Introduction

Every day, human innovation and creativity bring new advances. Adopting new technologies is the result of innovation and is not easy [1]. Rapid development and the need for scaled education in computer science in current time has made this field one of the innovative fields in education. To facilitate creative problem-solving, the learning process must be supported with the appropriate training and tools [2]. It is useful to study the options that are available to learners, in order to better understand their needs [3].

Computational thinking (CT) denotes the thought processes used for breaking down problems and formulating their solutions in a way that can be carried out through computational steps and algorithms [4]. It is envisioned that CT will become a fundamental component of every student’s learning [5]. Nonetheless, the incorporation of CT into, for instance, K-12 curricula and computer science education has encountered a variety of obstacles [6]. Despite these challenges, the benefits of integrating CT into education are manifold, particularly in relation to students’ perceptions of the digital tools they use. By fostering a CT mindset, students can gain a deeper understanding of the underlying principles of these tools, which can enhance their ability to use them effectively.

For instance, when students understand the algorithmic logic behind a programming language or software, they can better anticipate its behavior and troubleshoot issues. This not only improves their proficiency with the tool, but also their confidence in using it, which can positively impact their learning experience. While programming is not considered part of CT, it is the main approach to implement the solutions and algorithms created.

Computer programming is taught at universities and high schools all over the world [7]. Teaching introductory programming courses at the undergraduate level is a challenge. There are several reasons for this, which include a lack of previous knowledge of logic, proper tools, motivation to learn, etc. [8].

In order to support the teaching of programming, many methodologies, tools and techniques, and special languages have been proposed [9]. A collaborative learning tool where students work together toward achieving a learning goal can be one example of these tools [10]. As the teaching and learning process takes place in different contexts, there are still no standardized tools that can be used generically for learning programming. For the creation of efficient and standardized tools, we need studies on students’ perspectives to collect their opinions on different options as they will be the primary users of these tools. Studies on different user groups may produce comparative results where we can discover nuances between different students. The degree of trust that users have in innovative tools is another domain to explore. Do users like innovative tools for learning? How many of them trust tools for learning programming compared to learning by reading texts and learning from classroom lectures? All of these issues still need to be addressed in more detail.

To address these issues, we investigated the following research questions:

  • Q1: What concerns do students have regarding digital tools in their programming courses?

  • Q2: What are students’ perceptions regarding the usefulness of different digital tools in their programming courses?

  • Q3: What are students’ perceptions regarding the difficulty of different tasks in their programming courses?

We have collected data to answer the above research questions from two different parts of the world: Taiwan and Sweden. We were also interested in looking at our data from gender perspectives and to see if there are gendered differences in perceptions.

Section 2 reviews the literature on this topic and discusses how to involve the users’ views in the design process of applications, using a general framework. We also discuss some of the most common digital tools in programming courses in that section. In section 3, we describe the qualitative and quantitative methods that we used to analyze our data in detail. Section 4 provides our results and points to some notable findings among them. Section 5 revisits the research questions and answers them based on both our own findings and previous related research. Finally, Sections 6 and 7 conclude our paper and provide some future directions for our work.

2 Related work

This section dives into studies about the commonly used digital tools in teaching programming, students’ perspectives on these programming tools and User Centered Design in education. Despite numerous available tools, their efficacy varies among students, and some are scarcely utilized due to a lack of broader adaptability.

2.1 Common digital tools in programming education

Programming tool related issues rank among the top three subjects in the teaching and learning of programming [11]. Research projects devoted to tool development for programming education date back to 1984 [12]. Programming tools are typically designed for professionals and can be overwhelming for beginners due to their complexity and hard-to-understand error messages. To mitigate these issues, and to reduce and simplify teachers’ workloads, efforts have been directed towards the development of tools specifically tailored for novice programmers [13]. In most of studies in this field, the terms “tool” and “instrument” commonly denote software, applications, and programming environments. Nonetheless, researchers might also employ alternative phrases like technology, management methods, research tools, and teaching approaches interchangeably [14].

There are multiple categorizations of these tools in the literature. Pears et al. [13] have grouped them into Visualization tools (software applications that offer visual depictions of code execution, data structures, and program behavior, aiding students in comprehending intricate programming concepts [15] e.g. Jeliot [16], jGRASP [17], MatrixPro [18]), Automated assessment tools (software applications that automatically evaluate and provide feedback on students’ programming assignments and exercises [19] e.g. WebCat [20], CourseMaker [21]), Programming Environments (e.g. BlueJ [22], Dr. Java [23], Alice [24]), and Other tools [13]. They also point to intelligent tutoring systems in their “other tools” category, which nowadays constitute a notable part of the research in programming tools. Pillay has done a survey on applications of intelligent tutoring systems for novice programmers [25].

Another study categorized and listed a number of tools for teaching introductory programming courses [26]. The authors discussed different features of tools for teaching and learning introductory programming courses. This study was conducted by collecting opinions from an experienced community that were familiar with the tools, both regarding their experiences and regarding possible experiences.

Saito et al. [27] have proposed a taxonomy of programming learning tools, selecting 43 different tools based on their relevance to their keywords, including “Programming education tool kids” and “Programming learning tool kids”. These tools were evaluated based on 11 distinct features, such as which elements of games they have implemented, the type of interface they provide, etc. This detailed classification based on features has given a detailed understanding of the programming education tools available. It could inform the development of future tools and better teaching methods by emphasizing the pros and cons of existing tools.

Utilizing social networks is an innovative strategy being employed to enhance programming education. Educational social networks, such as Edmodo [28] or Schoology [29], provide a platform for students and teachers to share resources, discuss topics, and collaborate on projects, fostering a community of learning. They can be particularly effective in programming education, where peer-to-peer learning and collaboration are key. Beside these educational social networks, nowadays a lot of communications regarding the course materials happens in common social networks/Instant Messaging applications. Researchers in [30] has investigated effectiveness of Facebook as an instructional or course tool. Similar studies have been done about perceptions and attitudes of students about What’sApp [31] and Telegram [32]. In our work we have compared the perceptions about social networks with other available solutions to understand students’ perception better.

Gamification is also one of the innovative digital approaches towards education. It involves the application of game-design elements in non-game contexts, like health or education. Platforms like CodeCombat [33] or Codewars [34] gamify programming by turning coding exercises into challenges or games, making the learning process more engaging and enjoyable.

Another interesting study for us was [35] where researchers focused on secondary schools in South Africa for novice programming students. They have discussed programming assistance tools (PAT) and their applications for better teaching and learning processes. The research results are analyzed for improvements that could enable PATs to better support novice programmers and student communities.

Different elements of information technology (IT) have been analyzed for their relevance to educational methods that develop programming competencies and confidence in learners, including fragmentation of the programming process into small parts [36], visualization [37,38,39], and constructive [40] and think-aloud approaches [14, 41].

Pears et al. in [13] point to an interesting phenomenon: A meta-analysis of SIGCSE technical symposium proceedings [12] noted that 22% of papers presented at SIGCSE Technical Symposium conferences between the 1980s and the early 2000s were dedicated to the development of software tools for educators and students in CS1 and CS2. Despite this, only a few tools have seen widespread usage outside their institutions of origin, primarily because they were designed to solve local issues and were difficult to adapt for broader needs. Often these tools are prototypes developed through research funds or doctoral projects, which may lack the necessary support for adoption in diverse institutions after the initial developer moves on. Furthermore, research funding seldom supports customization and generalization of tools for broader usage. However, there are instances where tools such as the BlueJ programming environment [22], Karel family of microworlds of robots [42], and the CourseMarker automatic assessment tool [21], developed by larger groups, have seen considerable adoption and influence on teaching practices.

2.2 Students’ perceptions of tools used in programming education

Many studies have been performed on students of programming courses to understand their needs and thoughts about tools for teaching and learning. Research that was conducted on 38 North American students in [43] shows how cultural practice can be useful for learning computational tools. The authors found cultural biases in the structure of computational tools. They also proposed a design for computation tools that are culturally responsive rather than culturally biased.

Ortiz et al. [44] presented mobile learning tools to help programming students and their instructors. These tools are designed both for the teachers and the students. The teachers can use the tool to provide new exercises to the student, and students can submit solutions to the exercises using the mobile application. The proposed application was tested in two different introductory programming courses to observe the behavior of interaction between the students, teachers, and the application itself. They reported positive results, with an overall score of 4 out of 5 on Likert’s usability scale, and their conclusion was that students show interest in the broader introduction of such applications, noting that the ease of use and the possibilities offered by digital technologies should be considered to develop new, effective, and engaging learning methods. The authors also emphasize automatic assessments in these applications: “Without the automatic assessment, the tool would be worthless.” They suggest prioritizing providing more feedback for immediate assessment of a student’s work as one of the students’ requests about the process.

A study on participants in a massive open online course (MOOC) provided by Korea University about “Computational Thinking” has provided some insights about differences in perception of fairness and privacy when dealing with AI systems among males and females undergraduate students. The conclusion drawn is that female students exhibit more stricter views on fairness compared to male students, aligning with findings from prior research [45, 46]. The suggestion is that females tend to have a heightened sensitivity to privacy issues compared to males. However, given that some studies, like [47], indicate no significant gender differences in privacy concerns related to data collection and secondary use, further research is deemed necessary [48]. They also suggest that females have a more sensitive attitude toward privacy than males but as some researches show no gender differences in terms of privacy concerns regarding data collection and secondary use they suggest additional research is required [48].

Another team of researchers [49] presented comparative results on students’ motivation in introductory computer game programming courses in high school. The researchers collected data from 591 students to understand the students’ trend of motivation and attainment to course’s material. They found that there were significant differences in both motivational and attainment levels between different tools for programming education.

A study of 182 programming students in their 2nd semester at the University of Zagreb was performed in 2009 [50]. That study aimed to identify ways to design standardized tools for teaching introductory programming courses. The researchers suggested that due to a lack of previous knowledge and proper tools and guidelines, beginner students may conclude that programming is a challenging course. Better interactive tools can improve the attitudes of novice programming students.

Another study worthy of mention is [51]. In this study, students from six countries—Australia, Japan, Mexico, Sweden, Taiwan, and the United States—reflected on how their best and worst professors utilized different communication methods linked to effective teaching. This was an interesting study for us, as it compared different educational settings around the world in terms of students’ perceptions. Across cultures, students self-reported that their most favored professors were observed to use greater nonverbal expressiveness, relaxed movement, and engage in more in-class conversation and out-of-class communication compared to their least favored professors. Another interesting study in understanding cultural effects in digital tools is [52], where a review of the current research is presented with a focus on distance learning.

2.3 User centered design

The concept of User Centered Design (UCD), which puts user requirements at the forefront of each design process, dates to the works of Norman [53] in the 1980s. UCD is a repetitive process where designers engage with users via various methods throughout the design process, aiming for products that are highly usable and accessible to the final users [54]. Application of this scheme in education has also been a point of interest for Norman. Here, the focus is on the needs, skills, and interests of the learner. In [55] Norman and Spohrer present three dimensions for evaluating user-centered design in educational applications: Engagement, Effectiveness and Viability. The learner-centered approach is found to be particularly strong in engagement.

The systematic literature review detailed in [56] offers an overview of state-of-the-art applications of UCD for designing the UIs for e-Learning systems. It includes 17 works of literature from 2015 to 2020 reviewing how UCD was incorporated in e-Learning system development and its effect on system usability. It highlights that questionnaires, interviews, high-fidelity prototyping, and usability testing are the most commonly used methods in applying UCD. It was observed that projects employing UCD with user involvement across multiple phases often resulted in designs with high usability. Moreover, the study recommends involving both learners and subject matter experts in the e-Learning system design process to achieve superior user interfaces and effective learning experiences.

The authors of [57] provide a study on how to implement user centered design for programming learners. An interesting finding in their work was that “although visualized analytics increases the engagement of the students, a known design challenge arose from students’ feedback: how to design adaptive systems that would not add further complexity to the system by making the system more interactive?”. The study authors proposed that the implementation of personalization features in online learning environments needs to be smooth and not visible to users to address this issue.

3 Methodology

3.1 Survey design and structure

In this study, we utilized an online survey containing open-text and multiple-choice questions. The primary goal of our survey design was to gather data that could help us explore students’ perceptions and experiences of the application of digital tools in their programming courses. We were also interested in students’ own input regarding their concerns and suggestions on what they thought was unnecessary but included and what was necessary but not included in their courses.

Our survey included a combination of question types that allowed for both quantitative and qualitative analysis of the responses. The multiple-choice questions utilized a 5-point Likert scale format, ranging from options such as “Very helpful” to “Very unhelpful” or “Most Challenging” to “Least Challenging” depending on the context of the question. We also provided the participants with a “Not Applicable” option, except for the question about their concerns, as we wanted to push them to express their concerns as much as possible.

The survey questions were designed to answer the research questions explicitly. To explore research question Q1—about students’ concerns regarding computer-based technologies in their programming education—we inquired: “Rank the following options according to how problematic you find them in education from the most problematic to the most manageable.” The options provided to students were “Your privacy being violated”, “Users being unable to use them in the correct ways”, “Fairness of the decisions made by them”, “Adding unnecessary complexity to existing settings” and “Giving you improper recommendations that lead you to a wrong path”. Responses to these questions provide insights into how students prioritized the possible issues that might arise from the use of the technologies being studied. We also included an open text question to elicit any other concerns that students might have that we did not mention.

The question “What features and new technologies do you find useful in a programming course?” addresses research question Q2, about students’ perceptions regarding the usefulness of different digital tools and applications in their programming education. These options included “A social network for discussions and finding answers to your questions and proper materials related to your course”, “Using gamification to provide you materials and road maps”, “Using a recommender system to provide you with proper reading and watching materials”, “Using an automated recommender system to provide you projects and exercises”, “Usage of an automatic grading tool for your codes”, and “Using more visualization and sensible approaches to grasp the concepts”. To answer this research question, we also had two open text questions in which we asked students about what they thought was in their courses that did not need to be included and what they thought was lacking in their courses.

To delve into research question Q3—on students’ perceptions of the challenge level associated with various tasks in their programming education—we asked: “How challenging were the tasks that you encountered during your courses? Rank the following tasks from the most challenging to the easiest ones”. The tasks were “Solving your assignments during courses”, “Understanding the topics”, “Planning your schedule for studying the course materials and solving assignments”, “Interacting with other members in your groupworks”, “Interacting with your assistances and lecturers”, and “Finding proper learning materials based on your level”. We also asked students for the topic they found most challenging to grasp in their courses in an open text question.

We conducted this study for a number of reasons. First, our research is in the same domain as [50], but that study is quite old, and we need more recent data for a better understanding of students’ opinions about emerging solutions. Also, our research results consider a comparative analysis between two geographically distant countries.

Our research also differs from the contributions in [26] where the authors discussed different features of tools. This study was performed on an experienced community who were familiar with those tools, and their opinions were collected for the improvement of existing options. We have conducted our studies on the student population rather than more experienced groups to understand their perspectives.

3.2 Data collection

The survey was conducted on undergraduate computer science students from Sweden and Taiwan, focusing on those who were enrolled in the early stages of their programming courses. An online survey was utilized to gather data. The survey was presented in English for both locations, however, students had the flexibility to respond to open-ended questions in their language of choice. The rationale for selecting these two geographically diverse locations was dual: firstly, to enable comparative analysis in different environments (in comparison with the work in [43] where the study was performed within the student communities in North America), and secondly, due to the practicality of accessing CS course students in these regions. Two of the authors were students in the institutions that we have gathered the data from and we chose the courses based on their students relevance to our study. The intention was to get responses from students who are early in their encountering with programming. Survey was disseminated via email, and we also had the chance to introduce the study to two student groups during lectures in Sweden and Taiwan. Participation was voluntary and anonymous. In total, the survey was sent to 463 students and we received responses from 140 students, predominantly aged between 18 and 25 (Table 1). To encourage participation, Taiwanese students were offered a nominal credit towards their course grade which helped a lot with their response rate.

Table 1 Distribution of survey participants by gender and country
Full size table

3.3 Analysis

3.3.1 Quantitative analysis

In our study, we employed a quantitative analysis to understand the impact and perception of digital tools in programming courses across groups. One part of our quantitative analysis involved the calculation of frequency distributions and percentages to understand the distribution of responses for each question across groups. This is a common approach in quantitative research that allows for the summarization of large amounts of data, and the comparison of responses across groups [58].

The responses are mapped to an ordinal scale, representing levels of difficulty or effectiveness, depending on the question. This allows us to quantify the responses and perform a more nuanced analysis of the data. The use of ordinal scales is a well-established method in educational research, providing a way to rank-order data points within a variable [59].

The results of our analysis are visualized using stacked horizontal bar charts, which illustrate the distribution of a categorical variable across different groups [60].

The other part of our quantitative analysis involved employing a correlation analysis to examine the relationships between different variables derived from our survey data. This quantitative method allows us to measure and interpret the degree to which two variables are related to each other. To simplify the analysis, we have not calculated a separate value for each group of students; instead, the values are for aggregated groups of all students.

Correlation analysis can reveal patterns and associations that might not be immediately apparent. To facilitate correlation analysis, we align the scales for two different Likert scale categories by mapping the responses to a common ordinal scale. This allows us to compare and correlate variables from different categories, thereby expanding the scope of our analysis. However, it is important to note that this approach assumes that the intervals between points on the Likert scale are equal, which may not always be the case. This is a common limitation in the analysis of Likert scale data and should be taken into consideration when interpreting the results. Furthermore, while correlation analysis can reveal relationships between variables, of course it does not imply causation.

Through this quantitative analysis, our study aimed to provide valuable insights into the use and perception of digital tools in programming courses. The findings can inform the development and implementation of educational technology strategies that cater to the needs and preferences of diverse student groups.

3.3.2 Qualitative analysis

For qualitative questions, we have used inductive content analysis to categorize the answers. We conducted our analysis in three phases, namely preparation, organisation, and reporting phases [61]. This framework allowed us to find opinions from students, and it will help us to identify students’ perspectives on what they expect of technology that is used for programming education.

In the preparation phase, we defined individual students as the unit of analysis, and the goal was to identify similar themes or categories that students have mentioned. Data used for this part were students’ input on open-text questions about what they thought was unnecessary and what was not provided in their courses, along with their concerns and the topics they found hardest to grasp. In the organizing phase, we simplified the responses into different topics, or so-called “codes”, using an open coding technique [61]. Then, we grouped data to reduce the number of categories into broader higher-order categories [62,63,64]. We also made sure that each topic only belonged to the category to which it was assigned. In the reporting phase, we related the difficulties that students mentioned with the necessary technology purposed by them to address those problems.

We used inductive content analysis to develop a theory about the underlying structure of student experiences evident in the text (raw data). Inductive approaches are designed to help understand complex data by summarizing themes or categories from the raw data (a process also known as data reduction) [65], which suited our aims.

In total, 12 categories and 3 themes were found for the respective questions in the study. Themes in this context have a wider domain than categories. For the question about missing elements, five distinct categories emerged: “The need for more visual and interactive demonstrations”, “A reliable automatic grading tool”, “Clear instructions about course tools”, “Better topic-exercise connections”, and “Modern tools integration”. For the question on undesired elements, responses fell into two categories: “Unnecessary or advanced topics” and “Overly difficult assignments”. Additionally, students expressed concerns in three themes: “Bias”, “Difficulty adapting to new technologies due to insufficient guidance”, and “Privacy issues”. When asked about challenging topics in their courses, responses were categorized into five categories, including “Programming languages’ syntax, data structures, and compiling”; “Programming logic”; “Mathematical concepts”; “From learning to doing”; and “Debugging”.

4 Results

4.1 Quantitative results

The first set of graphs (see Fig. 1) illustrates the distribution of responses for each question related to the challenges faced by different groups in programming courses. These graphs highlight the similarities and differences in perceptions and experiences among the student groups (female, male, Swedish, Taiwanese). Here we can spot patterns such as Swedes’ lower degree of difficulty in interacting with other members, and how solving the assignments is perceived as the most challenging task among students.

Fig. 1
figure 1

Different student groups, perception about difficulty levels of different educational tasks

Full size image

The second set of graphs (see Fig. 2) provides a comparative analysis of students’ perceptions of the usefulness of various digital tools in their programming courses, across our four student groups. These visualizations highlight the varying degrees of perceived effectiveness for each technology, such as social networks for discussions and gamification for study materials. It can be seen that among the introduced technologies, the most useful perception by students is for visualisation. Another observation in this part of the data is Swedes’ more tendency in using “I don’t know” option compared to Taiwanese students.

Fig. 2
figure 2

Different student groups, perception about the usefulness of common digital tools used in programming education

Full size image

The third set of graphs (see Fig. 3) provides a view of the perceived challenges associated with digital tools in programming courses across our four student groups. These challenges, including unnecessary complexity, improper recommendations, privacy violations, user incompetence, and fairness of decisions, are evaluated on a scale from “The most manageable” to “The most problematic”. It can be seen in the data that women are more concerned about their privacy being violated and about being unable to use the technologies in the correct way compared to men. Additionally, for all the mentioned concerns, Swedish students chose the “most problematic” option more frequently than the Taiwanese students.

Fig. 3
figure 3

Different student groups, perception of concerns about using digital tools in programming education

Full size image

Figure 4 is a heat-map visualization generated from parts of the quantitative data, underscoring relations between our quantitative variables. These variables encapsulate factors such as personal study management, interaction within academic communities, selection of appropriate learning materials, use of educational technology, and other related challenges. The heatmap, which was derived using Pearson’s correlation coefficient method, offers an illustrative interpretation of the interplay among these variables. By observing the correlation strength and direction among the factors, this visualization can serve as a guide for understanding the dynamics of student learning experiences. Consequently, the analysis reveals where interventions could be most beneficial and aids in devising strategies to mitigate potential impediments.

Fig. 4
figure 4

Correlations between student responses on the difficulty level of tasks, usefulness of different digital tools and associated concerns

Full size image

Some of the notable correlations that can be read from this graph are:

  • There is a relatively strong positive correlation between perceived level of difficulty in solving the assignments and planning the schedule (correlation value = 0.38) and with understanding the topic (correlation value = 0.24).

  • Perceived level of difficulty in Interaction between students with assistants and lecturers is positively correlated with the perceived level of difficulty in Interaction between students with other students in their groupworks (correlation value = 0.3).

  • Perceived level of difficulty in finding proper learning material is positively correlated with the perceived level of difficulty in interactions between students and other student members of their groupworks (correlation value = 0.48; this is the most notable correlation found).

  • Perception of usefulness for a social network during programming education is negatively correlated with perception of difficulty in solving the assignments (correlation value = − 0.3).

  • The level of concern for privacy is positively correlated with the perception of usefulness for a recommender system for watching and learning materials (correlation value = 0.3).

  • The level of concern for fairness is positively correlated with the level of concern for privacy (correlation value = 0.35) and also the level of concern for being able to use the technology in the proper way (correlation value = 0.32). It is also positively correlated with the perception of usefulness for recommender systems (correlation value = 0.3).

  • The level of concern for adding unnecessary complexity is in positive correlations with the level of concern for giving improper recommendations (correlation value = 0.33) and being unable to use the technology (correlation value = 0.35).

4.2 Qualitative results

Our study included two open-ended questions to gather students’ perspectives on the learning tools and technologies that were present but unnecessary, as well as elements that were absent but desired. Subsequently, we used content analysis to process and summarize the responses into key facets of effective computing education as per students’ opinions.

For the question: “What was lacking in the course’s setting regarding tools and technologies?”, we have categorized the responses as the following. Some excerpts of students’ responses have been provided after each category.

  1. 1.

    More visual aids, animations, or interactive demonstrations to understand concepts better

    “Too little interacting tools to understand the topics.”

    “It would have been helpful to have more visualization.”

    “Animations instead of still images, to show how a certain algorithm or data structure works.”

  2. 2.

    An automatic grading tool for code assignments and practices

    “Usage of an automatic grading tool for our codes for homework, so that we can check our answers while doing homework.”

    “An automatic grading system (Online judge system)”

  3. 3.

    Clear instructions about the tools used in the course

    “A guide of the tool”

    “Some proper introductions to each tool”

  4. 4.

    Provision of better connections between different topics and exercises

    “Connection of different topics”

    “Practical exercises to use the information that was learned”

    “Proper exercises. There was a gap between lectures and homework. I need more proper exercise to practice step by step.”

  5. 5.

    Modern tools and technologies

    “We worked with a fairly old graphics library during our C++ course, which should have been changed to a slightly more modern one as the challenges with this library are vastly different from what you may find with other, similar libraries”

    “Outdated information and not enough focus on good programming practice in the used language”

For the question “What should not have been included in the courses?” we identified with the following categories:

  1. 1.

    Certain topics or contents that were unnecessary or too advanced

    “I think they should not enforce us to learn some of the compulsory that the knowledge not everyone uses at all such as calculus”

    Too much knowledge that may not be usable for me. (Translated from Chinese)

  2. 2.

    Homework, assignments, and exams which were overly difficult or bothersome

    “Assignments that are too hard.”

    Homework’s difficulties and the course’s content do not match. Therefore, I become confused when learning. I thought that I understood the lecture, but it’s a whole other level in practical application. (Translated from Chinese)

For this part of the analysis, we conclude that “immediate feedback during programming”, “visualization of computer science concepts”, “provision of tutorials for programming tools”, and “provision of real-life applications of introduced concepts” are students’ prioritized issues among what they feel is missed or excessive in their programming courses.

We also had an open-text question about students’ concerns: “Do you have any other concerns regarding the use of new technologies in your programming courses?” For this part, the responses were categorized into three main themes, namely bias, difficulty, and privacy:

  1. 1.

    Some talk about how using AI or an automated grading system may misjudge or cause potential inequality due to biases in the training data.

    “It is very easy for AI to replicate unjust systems, my concern would be that it would systematically introduce inequality”

    “Given a biased set of training data could lead to a very VERY serious issue, for instance, we have seen AIs made by companies that have turned racist/fascist or straight up Adolf Hitler 2.0 due to the biased data given to them”

  2. 2.

    Some talk about how they need time to get familiar with these new technologies and suggest that the guidance is insufficient.

    “A guide of a tool is lacking in the course”

    Afraid of not knowing how to use (Translated from Chinese)

  3. 3.

    Some Swedish students mention privacy issues.

    “Privacy and integrity”

Through our content analysis on our open text question, “What was the most challenging topic to understand during your courses?”, we investigated some problems that students mentioned. Detailed categorization of the topics that students have mentioned can be found on Table 2. We categorized those responses into five categories:

  1. 1.

    Programming Languages’ Syntax, Data Structure, and Compiling

  2. 2.

    Programming Logic

  3. 3.

    Mathematical Concepts

  4. 4.

    From Learning to Doing

  5. 5.

    Debugging

Table 2 Inductive content analysis on “what was the hardest topic to understand that you encountered during your programming course?”
Full size table

Moreover, 67% of the students who participated in the course taught in C responded that the pointer was the most challenging topic in their course. This results in having a measure of 5.02 out of 6 for the difficulty indicator (mean score in mapped Likert scale) in solving the assignment in our quantitative data. For comparison, other students had a “3.53/6” difficulty indicator for that task.

5 Discussion

In this section, we revisit our research questions with the insights that we can get from our data, and discuss how these results relate to the findings of existing research as well as limitations with the study.

5.1 What concerns do students have regarding digital tools in their programming education?

Our first question was about concerns. From Fig. 3 it can be seen that Privacy and Improper Recommendations are the two most problematic issues.

There are several studies that show there is a gender difference in privacy concerns when using digital technologies, implying women are more concerned compared to men [66,67,68]. This difference is also visible in our data when we look at the relative responses of men and women in Fig. 3. Our interpretation of this phenomenon in the educational setting is close to the one provided in [69]: “... interpretation is that, compared to men, women feel more vulnerable, especially in terms of physical safety and psychological well-being (such as being stalked) and that women bring these concerns into a greater variety of roles in life”.

It is interesting to note that based on [70], men, although less concerned about privacy than women, tend to rely more consistently on technical means of protection regarding privacy. Figure 3 also shows that men are less concerned about whether they would be able to use educational technology in a proper manner (competency).

These data highlight the areas of concern that need to be addressed when developing and implementing digital tools. By understanding these concerns, educators and developers can tailor their strategies and tools to better address the concerns that could interfere with the application of these technologies among their intended users.

5.2 What are students’ perceptions regarding the usefulness of different digital tools in their programming education?

Our second research question was about perceptions regarding the usefulness of these tools. Figure 2 summarizes our quantitative results on this question. These data are useful in understanding cultural or gender-based differences in the perceived usefulness of these technologies. For example, almost 30% of Swedish students said that they didn’t know if gamification could provide them with useful course materials and help them with their plannings. The Swedish students also expressed more doubt of the utility of automated recommender systems than the Taiwanese students. Interestingly, Fig. 2 also shows how Swedes tend to use the “I don’t know” option more than Taiwanese students in general.

The impact of visualization in programming education is a relatively old idea. There have been plenty of papers specifically studying how and why visualization helps programming education [71]. As can be seen by our data in Fig. 2, this method is perceived as significantly more useful across all student groups.

In the open-text responses, many students expressed that immediate feedback during programming would be very helpful to assist students in learning programming. This helps students debug in an efficient way. As we can see in our data, debugging is considered one of the hardest tasks students encountered during programming courses. With regard to the properties of fast response and accessibility of educational technology, especially with the emergence of large language models that can understand code, educators and developers can easily develop learning tools that provide immediate feedback.

5.3 What are students’ perceptions regarding how challenging they find different tasks in their programming education?

Our third research question was about challenges students faced. The most notable pattern visible in our analyzed quantitative data for this question was about how challenging students felt their assignments were to solve.

Solving assignments is probably the most effort-taking task that students need to perform, but there are other aspects to why it could be perceived as challenging and how it impacts students.

Researchers in University of California San Diego carried out a survey on student experiences in introductory computing courses. They discovered that students with lower performance reported increased stress in various areas—cognitive, socio-economic, and personal—suggesting that struggling students frequently contend with multiple challenges. Additionally, it was observed that students without prior computing experience faced significantly more difficulties than those with previous experience, regardless of their exam performance levels. [72].

Support for students facing challenges should be multi-dimensional, extending beyond mere teaching techniques. For example, offering more flexibility in schedules and assignments might aid these students in handling stress while continuing their learning. Additionally, it’s been recommended to link financial aid to the difficulty level of courses instead of a fixed course count. Introducing computer science courses and camps at the K-12 level is also advised as a beneficial strategy. [73].

It is worth noting that solving assignments was not reported as being as difficult by Swedish students as by Taiwanese students. A study in 2007 on Singaporean Chinese engineering students hints at something interesting regarding the roots of this phenomenon: “Influenced by their national culture, which values achievement and success, the students cared less for fun and exciting activities and valued tasks that led to achievement in learning. Influenced by their ethnic (Chinese) culture, which stresses learning from an authority figure, the students preferred teacher feedback to peer feedback.” [52, 74].

Figure 1 also shows that Swedes reported more comfort in their interactions, both with other students and with teachers compared to Taiwanese students. Digging deeper into this, we encountered [51] indicating that students in Australia, Sweden, and the U.S. observed their “best professors” as having more in-class conversations, in contrast to students from Japan or Taiwan and their perceptions of their “best professors”.

Another comparison among cultures is made in [52] which differentiates between “Low Context” and “High Context” cultural groups in terms of their orientation towards computer-mediated communication (CMC). Students from “Low Context” cultures, described as “Outwardly Oriented,” utilize CMC primarily to consider and reflect upon others’ opinions. Conversely, those from “High Context” cultures, characterized as “Inwardly Oriented,” value CMC as an opportunity to focus more on their own contributions and reflections. Low context cultures, typically found in Western societies, prioritize individualism, direct communication, and outcome-focused learning. They encourage active student participation, fostering personal skills, and lifelong learning attitudes. The student-teacher relationship is informal, with teachers acting as guides or facilitators. Conversely, high context cultures, common in Eastern societies, emphasize collectivism, indirect communication, and input-focused learning. Students in these cultures are more passive, accepting, and oriented toward reproducing teacher-provided material. The student-teacher relationship is formal, with students accepting teacher knowledge [75].

5.4 Gender differences in classroom

Gender roles refer to the behaviors, attitudes, and expectations specifically linked to a particular gender. Psychologists make a distinction between gender differences, which are tied to societal roles, and sex differences, which pertain solely to physiological and anatomical aspects. In this context, our focus is on gender differences.

While there are numerous exceptions, on average, boys and girls exhibit differences that align with traditional gender stereotypes, influencing their actions and interactions in educational settings [76, 77]. These distinctions include physical behaviors, social interaction styles, academic motivations, and preferences [78]. Main influences on gender role differences stem from parents, peers, and media. Although teachers are not the main contributors to these disparities, their responses and decisions regarding students can reinforce these gender roles. It is important to note that the variations within each gender group are typically much greater than those between the two groups [76].

In classroom settings, variations in social interaction styles are evident. On average, boys tend to be more vocal during class discussions, often speaking out even if they haven’t been prompted or if they aren’t as knowledgeable about the subject as their peers [79]. When collaborating on a project within a mixed-gender group, they often overlook the input and contributions of girls. This dynamic mirrors interaction patterns observed in broader societal contexts, where men often disregard women’s remarks and contributions [76, 80].

Regarding differences among genders in our study, using Fig. 1, we can see that interactions between students and teachers were reported as easier by male students compared to female students. In existing literature, we found similar patterns in a study on a Christian middle school in southwestern British Columbia, Canada [81]. This study involved eight teachers from grades six to eight. Observations were made during two 20-min lessons for each teacher, categorizing each teacher-student interaction as academic or behavioral, and further as positive, negative, or neutral. The outcomes of this study indicated that boys received more teacher attention compared to girls in this school setting. Additionally, it was observed that girls had fewer behavioral interactions with teachers than boys.

While teachers typically aim to engage with both genders impartially and often achieve this goal, research indicates that there are instances where they may interact differently with boys and girls, possibly without being consciously aware of it. Seifert and Sutton [76] identified three types of gender differences in educational settings. Firstly, the overall amount of attention teachers give to each gender. According to their findings, teachers interacted with boys more frequently than girls, with a margin ranging from 10 to 30%, varying by student grade level and teacher personality [82]. The second difference lies in the visibility or “publicity” of conversations between teachers and students. The third difference is related to the types of behaviors that lead teachers to either support or criticize students.

5.5 Limitations

Our study is not without its limitations. Firstly, the data were collected from students in Sweden and Taiwan only. While these two countries provide a diverse range of responses, they most likely do not fully represent the global student population’s perspectives on computer-based technologies in programming education. Therefore, the findings may not be generalizable to all contexts.

Secondly, the use of Likert scale responses in our correlation analysis assumes that the intervals between points on the scale are equal. This is a common limitation in the analysis of Likert scale data, and it should be taken into consideration when interpreting the results. The ordinal nature of Likert scale data also means that it only provides information about the relative, not absolute, differences between categories.

Thirdly, while we have used inductive content analysis for the qualitative questions, the interpretation of the responses is subjective and dependent on the researchers’ perspectives. This could potentially introduce bias into the analysis.

Lastly, our study is cross-sectional, meaning that it captures a snapshot of students’ perspectives at a specific point in time. As such, it may not account for changes in students’ perceptions over time or as they gain more experience with programming and computer-based technologies.

Despite these limitations, we believe that our study provides valuable insights into students’ perspectives on computer-based technologies in programming education. These insights can inform the development of more effective and engaging digital tools for programming education.

It is also worth mentioning that the Taiwanese students received a nominal course credit for their participation, while the Swedish students did not. The effect of this can be seen in the high response rate from Taiwanese students compared to Swedish students in Table 1. Since the participation rate was 12% in Sweden, this means that there is a risk of a selection bias. As such, the comparison between Sweden and Taiwan should be interpreted with some care. However, since the questions in the survey are not directly related to performance in the particular courses, and since we see many similarities between the two countries, we believe that this is not a major issue.

In our review of the literature, we have not found any similar research to our work wherein a comparative study was performed on student communities from different parts of the world, and the data was analyzed to find out the students’ interests in tools for learning programming in their first encounters with programming in academic settings.

6 Conclusion

Through this study, we examined the challenges that students face while learning programming as well as their concerns and perceptions regarding various educational tools. Effective educational tools can reduce students’ stress and provide personalized support during their learning progress. This assistance can help students to meet the requirements more easily for solving programming problems. One of the key findings in our study was that complex topics, such as pointers, posed significant challenges for most students. We also found that students struggle with transforming their knowledge into actual programming.

The use of various teaching tools, techniques, and resources was discussed. The significance of peer interactions, both offline and online, was evident, highlighting the importance of collaborative learning environments in programming education. Immediate feedback during programming also emerged as a crucial expected tool that could assist students in navigating the challenges of debugging, one of the most difficult aspects of their courses. Additionally, while debugging and testing tools are undoubtedly crucial, their inherent complexities underscore the need for more beginner-friendly interfaces.

Visualization stood out as a tool that students saw as particularly effective for teaching abstract concepts in programming. The impact of this tool on students’ learning experiences underscores the need for educational resources that can simplify complex ideas, making them more comprehensible for students at all levels.

Moreover, our data revealed several patterns, including differing perceptions of comfort level in student interactions, gender differences in privacy concerns, and variations in perceived difficulty of assignment completion. Also, based on both our quantitative and qualitative research, we found that Swedish students are more skeptical towards computer-based educational solutions and more worried about privacy and integrity issues. These patterns underscore the need for a multifaceted approach in addressing students’ challenges, with a focus on inclusivity and flexibility.

The exploration of these challenges and potential solutions provides a better understanding of the current landscape of programming education, which can lead to more user-centered design for these tools. We believe that our study provides valuable insights that can inform the development of more effective and engaging digital tools for programming education. It is worth mentioning that as we adapt teaching methods to meet students’ needs, we must remember that the field of computer science is continually evolving. This demands continuous research and flexibility in our educational approach.

In the next section, we will look at some potential future developments and directions in understanding students’ perspectives towards educational tools in programming courses.

7 Future works

Future research can expand on our study by:

  • Including a different and preferably more diverse sample of students to acquire a more holistic understanding

  • Using a longitudinal design to track changes over time, which could provide deeper insights into the long-term efficacy of various learning tools and teaching strategies, as well as the evolving landscape of programming education

  • Employing additional quantitative and qualitative methods that offer a more complete insight into students’ viewpoints

We also suggest further investigation from the user’s side into the role of AI in grading and feedback. As highlighted by students’ concerns, automated grading systems could introduce bias and potential inequality. On the other hand, nobody wants to miss out on the opportunities for providing fast feedback, better clarification on subjects, and more customized education tailored to the final users that AI systems can bring. As pointed out in the discussion section, one of the major applications of large language models in the field of programming education is their capacity to understand students’ code and provide them with immediate feedback and suggestions. Future efforts should consider developing more transparent, equitable, and privacy-aware AI systems for education.

We look forward to exploring these research areas in our future work, aiming ultimately to enhance the quality and effectiveness of computer programming education