1 Introduction

There is a massive shortage of skilled software developers in this age of digitalization. According to Gartner, the demand for IT professionals will be multiple times more than supply (Waszkowski 2019; Torres 2018). To make matters worse, training and hiring new software developers are very expensive in this rapidly evolving world. LCSD aims to address this issue by democratizing software development to domain experts and accelerating the development and deployment process. It tries to bridge the gap between the system requirement and the developer constraints, which is a common reason for long development times in complex business applications.

LCSD is a novel paradigm for developing software applications with minimal hand-coding through visual programming, a graphical user interface, and model-driven design. LCSD embodies End User Software Programming (Pane and Myers 2006) by democratizing application development to software practitioners from diverse backgrounds (Di Sipio et al. 2020). It combines various approaches such as visual modeling, rapid app development, model-driven development, cloud computing, and automatic code generation. Low-code development tools enable the development of production-ready apps with less coding by facilitating automatic code generation. Additionally, LCSD platforms also provide more flexibility and agility, faster development time that allows responding quickly to market needs, less bug fixing, less deployment effort, and easier maintenance (Sahay et al. 2020; Di Sipio et al. 2020). These platforms are used to develop high-performance database-driven mobile and online applications for various purposes. As a result, low-code development is rapidly growing in popularity. According to Forrester, the LCSD platform market is estimated to reach $21 billion by 2022. By 2024, over 65% of big companies will utilize LCSD systems to some extent, according to a Gartner report (Wong et al. 2019).

To date, there are more than 400 LCSD platforms (2022), offered by almost all major companies like Google (2020) and Salesforce (2022). Naturally, LCSD has some unique challenges (Sahay et al. 2020). Wrong choice of LCSD application/platforms may cause a waste of time and resources. There is also concern about the security/scalability of LCSD applications (Khorram et al. 2020). With interests in LCSD growing, we observe discussions about LCSD platforms are becoming prevalent in online developer forums like Stack Overflow (SO). SO is a large online technical Q&A site with around 120 million posts and 12 million registered users (Overflow 2020). Several research has been conducted to analyze SO posts (e.g., IoT (Uddin et al. 2021b), big data (Bagherzadeh and Khatchadourian 2019), blockchain (Wan et al. 2019) concurrency (Ahmed and Bagherzadeh 2018), microservices (Bandeira et al. 2019)). The studies, however, did not analyze discussions about LCSD platforms in SO.

In 2021, we conducted an empirical study (Alamin et al. 2021) by analyzing 4,785 posts (3,597 questions + 1,118 accepted answers) from SO that contained discussion about nine LCSD platforms. The study offered, for the first time, an overview of the challenges software developers face while using LCSD platforms. However, to date, there are over 400 LCSD platforms and we observed discussions about many of those platforms in SO. Therefore, it was important that we revisit our empirical study with a larger dataset of discussions about LCSD platforms in SO. In addition, given that the previous empirical study was a conference paper, the analysis was not as in-depth as we could have provided due to space limitations. Therefore, a larger-scale empirical study of the challenges developers face to adopt and use the LCSD platforms was warranted. Such insights can complement our previous empirical study (Alamin et al. 2021) as well as the existing LCSD literature – which so far has mainly used surveys or controlled studies to understand the needs of low-code practitioners (Fryling 2019; Kourouklidis et al. 2020; Alonso et al. 2020; Khorram et al. 2020).

Specifically, in this paper, we present an empirical study of 33.7K SO posts relating to the top 38 LCSD platforms (according to Gartner (2022)) at the time of our analysis to ascertain the interest and challenges of LCSD practitioners. We answer five research questions by analyzing the dataset.

RQ1. What topics do LCSD practitioners discuss? :

Given that LCSD is a novel paradigm, it is vital to study the types of topics discussed by LCSD practitioners on a technical Q&A platform such as SO. As a result, we use the topic modelling method LDA (Blei et al. 2003) on our 33.7K post dataset. We find a total of 40 LCSD topics grouped into five categories: Application Customization (30% Questions, 11 Topics), Data Storage (25% Questions, 9 Topics), Platform Adoption (20% Questions, 9 Topics), Platform Maintenance (14% Questions, 6 Topics), and Third-Party Integration (12% Questions, 5 Topics). Around 34% of questions are particular to the many supported capabilities of LCSD platforms, while the remaining 66% are regarding development activities, namely application customization. This is because the LCSD platform’s features are naturally oriented around a graphical user interface (GUI) in a drag-and-drop environment. As a result, any customization of such features that are not native to the LCSD platforms becomes difficult.

RQ2. How do the LCSD topics evolve over time? :

We elaborate on our findings from RQ1 by examining how the observed LCSD topics evolved in SO over time. We conduct an in-depth analysis of LCSD-related discussions from 2008 to mid-2021 in SO. We discover that since 2012, discussion about LCSD has piqued community interest, which has increased significantly throughout the pandemic, i.e., since 2020. In recent years, Platform Adoption-related discussions have acquired more traction than original application customization or database query-related discussions. Future research and LCSD platform vendors should support emerging topics such as Library Dependency Management, External Web Request Processing, Platform Infrastructure API, and Data Migration.

RQ3. What types of questions are asked across the observed topic categories? :

From RQ1, we find some of the unique challenges for LCSD practitioners regarding Customization, Data Storage on the completely managed cloud platforms. This motivates us to explore further to understand more of those challenges. For instance, we want to understand if practitioners mostly ask about different solution approaches (i.e., How-type) or further explanation clarification type (Why/What-type). Following previous studies(Uddin et al. 2021b; Abdellatif et al. 2020), we manually annotated a statistically significant number of posts (e.g., 471 Questions) into four categories. We find that How-type (57%) is the most common form of inquiry across all five topic categories, followed by What-type (18%), Why-type (14%), and Other-type (12%) questions. Most of the How-type questions are application implementation-related, and most of the What-type and Why-type Questions are server configuration and troubleshooting related. According to our findings, proper documentation and tutorials might significantly reduce these challenges.

RQ4. How are the observed topic categories discussed across SDLC phases? :

Our findings from the previous research questions examined the practitioners’ challenges on LCSD platforms and their evolution. The acceptance of this emerging technology depends largely on effective adoption into the various stages of a software development life cycle (SDLC). So, following our previous study (Alamin et al. 2021) we manually annotate statistically significant samples (e.g., 471 Questions) into six agile SDLC stages. We find that the Implementation (65%) is the most prominent phase in terms of the number of questions, followed by Application Design (17%) and Requirement Analysis & Planning (9.1%).

RQ5.What LCSD topics are the most difficult to answer? :

LCSD practitioners face many different challenges to understand different features of the cloud platform, server configuration. LCSD vendors aim to provide support from requirement gathering to deployment and maintenance, but practitioners still struggle with customization, data management, and cloud configuration. We find that, while the topic of application customization and the Implementation-SDLC are the most prevalent, Platform Adoption topic category and the Deployment-SDLC and Maintenance-SDLC as the most popular and hardest to get accepted answers.

This paper extends our previous paper (Alamin et al. 2021) along two major dimensions: the data used and the results reported. We offer details about the extensions below.

  1. 1.

    Data (see Section 3). The dataset in this study is significantly larger and more diverse than our previous paper as follows.

    • Size. The size of the SO dataset in this paper is almost seven times bigger than the dataset used in our previous paper. This study analyses 33766 posts (26763 Questions + 11355 Accepted Answers). Our prior paper examined 4,785 posts (3597 Q + 1188 A).

    • Time. This study analyzesLCSD-related discussions in SO between July 2008 to May 2021, while the previous study analyzed the discussions between July 2008 to May 2020.

    • LCSD Platforms. This study analyzes 64 LCSD-related tags which contain 38 LCSD platforms, while the previous study analyzed 19 SO tags related to 9 LCSD platforms.

  2. 2.

    Empirical Study (see Section 4). This paper considerably enhances our understanding of LCSD platforms over our previous paper (Alamin et al. 2021) as follows.

    • Research Questions (RQ). We have answered five research questions (RQ2, RQ3) in this paper compared to three RQs in our previous paper (RQ1, RQ4, RQ5). The two new RQs offer insights on the type of LCSD questions asked and the evolution of the LCSD topics. Our revision of the previous three RQs provided several new results as follows.

    • LCSD Topics. In this study, we found 40 topics organized into five high-level categories. We found 13 topics organized into four high-level categories in our previous paper. While we found all the previous 13 topics, we also found 27 new LCSD topics. This study found Platform Maintenance as a new high-level topic category (see Section 4.1).

    • Finer-Grained Analysis. Due to our use of more data, we find better results from our topic modeling. For example, some topics from our previous studies are broken down into more informative/coherent topics. For example, Client-Server Communication and IO from Platform Adoption topic category became topics Web Service Communication and Message Queue under Asynchronous Service to Service Communication sub-sub-category in this study as those topics contained more coherent discussions. Similarly, we have expanded our understanding of software development lifecycle phases (SDLC) around the new 40 topics (see Section 4.4).

    • Topic Evolution. Our new RQ2 analyzes the evolution of the observed LCSD topics in Section 4.2. We further discuss the prevalence and evolution of the topics across the top 10 LCSD platforms in our dataset (see Section 5.4).

    • Question Type. Our new RQ3 offers insights into the type of questions asked across the observed LCSD topics (see Section 4.3).

    • Popularity vs Difficulty. In addition to analyzing the popularity and difficulty of all 40 topics in Section 4.5, we also offer the following new insights. (a) Following a recent study (Uddin et al. 2021b), we report the popularity and the difficulty using two fused metrics (see Section 4.5). (b) We report the popularity and difficulty of the LCSD question types and SDLC phases (see Section 5.6)

  3. 3.

    Related Work. We have expanded our literature review with a comparison of key metrics around our observed LCSD topics against those previously reported for other domains while using the SO data (see Section 7.2).

Our study findings can enhance our understanding of the developers’ struggle while using LCSD platforms. The findings would help the research community and platform vendors better focus on the specific LCSD areas. The practitioners can prepare for difficult areas. LCSD platforms can design more effective and usable tools. All stakeholders can collaborate to provide enhanced documentation assistance. The LCSD vendors can support increased customization of the LCSD middleware and UI to make the provided functionalities more usable.

Replication Package: The code and the datasets generated during this study is available in our public GitHub repository https://github.com/al-alamin/LCSD_challenge_EMSE

2 Background

This section aims to provide a high-level overview of LCSD development, as well as some of the relevant technologies and research that have shaped this industry. We hope that this will serve as a resource for future researchers (particularly those interested in the underlying technologies)) and practitioners to learn and contribute more to this emerging new field.

Low-code Software Application To cater to the demand of the competitive market, business organizations often need to quickly develop and deliver customer-facing applications. LCSD platform allows the quick translation of the business requirement into a usable software application. It also enables citizen developers of varying levels of software development experience to develop applications using visual tools to design the user interface in a drag-and-drop manner and deploy them easily (Low-Code Development Platform 2022). LCSD is inspired by the model-driven software principle where abstract representations of the knowledge and activities drive the development, rather than focusing on algorithmic computation (Sahay et al. 2020). LCSD platforms aim to abstract away the complexity of testing, deployment, and maintenance that we observe in traditional software development. Some of the most popular low-code platforms are Appian (2022), Google App Maker (2020), Microsoft Powerapps (2022), and Salesforce Lightning (2022).

Technologies that Shaped LCSD Model-driven Software Engineering (MDSE) field proposes the adoption of domain-specific modeling practices (Brambilla et al. 2017). Low-code platforms adopt model-driven engineering (MDE) principles as their core that has been applied in several engineering disciplines for the purpose of automation, analysis, user interface design (Botterweck 2006; Pleuss et al. 2013; Brambilla et al. 2017) and abstraction possibilities enabled by the adoption of modelling and meta modeling (Basciani et al. 2014). Besides, End-User Development (EUD) is a set of methods, techniques, and tools that allow users of software systems, who are mainly non-professional software developers, at some point to create, modify or extend a software artifact (Paternò 2013; Fischer et al. 2004). EUD for GUIs can be a good example of its usage (Costabile et al. 2007). Scratch (Resnick et al. 2009), Bloqqi (Fors 2016), EUD-MARS (Akiki et al. 2020), App Inventor (Wolber 2011), AppSheet (AppSheet 2021) are such “low-code/no-code” application development tools that offer visual drag-and-drop facilities. Similarly, there are several other research areas within the domains of HCI (Sinha et al. 2010) and Software engineering, such as Visual Programming (Burnett and McIntyre 1995), Programming by example (Halbert 1984), End users programming (Myers et al. 2006), domain specific language (Mernik et al. 2005; Van Deursen et al. 2000), trigger action programming (Ur et al. 2014) that aim to enhance the technologies underlying low-code software development. Thus, gaining a better knowledge of the problems associated with low-code platforms through developer discussion would be extremely beneficial for further improving these studies.

Development Phases of an LCSD Application A typical LCSD application can be built in two ways (Sahay et al. 2020): 1. “UI to Data Design”, where developers create UI and then connect the UI to necessary data sources, or 2. “Data to UI” where the design of the data model is followed by the design of the user interfaces. In both approaches, application logic is implemented, and then third-party services and APIs are integrated. APIs are interfaces to reusable software libraries (Robillard et al. 2012). A major motivation behind LCSD is to build applications, get reviews from the users, and incorporate those changes quickly (Waszkowski 2019). Some software development approaches are quite popular and supported by different LCSD platforms, such as Iterative software development (Basil and Turner 1975) which is based on the iterative development of the application. In this way, every step is cyclically repeated one after another. In practice, this is very helpful because it allows developing and improving the application gradually. Another approach can be Rapid application development (RAD) (Beynon-Davies et al. 1999) is a software development methodology that promotes the rapid release of a software prototype. It is an agile approach and aims to utilize user feedback from the prototype to deliver a better product. Another popular methodology is the agile development methodology (Beck et al. 2001) which is a collection of approaches and practices that promote the evolution of software development through collaboration among cross-functional teams.

Different LCSD teams may adopt different SDLC approaches. However, we focus mostly on Agile methodology for this study because Agile and LCSD can go hand in hand because the fundamental principle and objective are customer satisfaction and continuous incremental delivery. Traditional software development teams widely use agile, which also provides the generalizability for other methodologies. So, in this study, we map agile software development life cycle phases with LCSD methodologies. The inner circle of Fig. 1 shows the important development phases of an LCSD application, as outlined in (Sahay et al. 2020). The outer circle of Fig. 1 shows the phases in a traditional agile software development environment. As LCSD platforms take care of many application development challenges, some of the agile application development phases have shorter time/execution spans in LCSD than traditional software development.

Fig. 1
figure 1

Agile methodologies in traditional vs LCSD development

Full size image

3 Study Data Collection and Topic Modeling

In this Section, we discuss our data collection process to find LCSD related posts (Section 3.1). We then discuss the details about our pre-processing and topic modeling steps on the selected posts (Section 3.2).

3.1 Data Collection

We collect LCSD related SO posts in three steps: (1) Download SO data dump, (2) Identify LCSD related tag list, and (3) Extract LCSD related posts from the data dump based on our selected tag list. We describe the steps below.

Step 1: Download SO data dump. For this study, we used the most popular Q&A site, Stack Overflow (SO), where developers from diverse background discuss about various software and hardware related issues (2020). For this study, We downloaded SO data dump (Exchange 2020) of May 2021 which was the latest dataset available during the starting of this study. We used the contents of “Post.xml” file, which contained information about each post like the post’s unique ID, type (Question or Answer), title, body, associated tags, creation date, view-count, etc. Our data dump included discussion of 12 years from July 2008 to July 2021 and contained around 53,086,327 posts. Out of them, 21,286,478 (i.e., 40.1%) are questions, 31,799,849 (i.e., 59.9%) are answers, and 51.5% questions had accepted answers. Around 12 million users from all over the world participated in the discussions.

Each SO post contains 19 attributes, and some of the relevant are: (1) Post’s body with code snippets, (2) Post’s Id, creation and modification time, (3) Post’s view count, favorite count, score, (4) User Id of the creator, (5) Accepted answer Id and a list of 0 to 5 tags.

Step 2: Identify low-code tags. We need to identify the tags that are related to LCSD in order to extract low-code related posts from SO discussions. To find relevant tags, we followed a similar procedure used in prior work (Alamin et al. 2021; Uddin et al. 2021b; Abdellatif et al. 2020; Ahmed and Bagherzadeh 2018; Wan et al. 2019; Linares-Vásquez et al. 2013). At Step 1, we identify the initial low-code related tags and call them Tinit. At Step 2, we finalize our low-code tag list following related work (Bagherzadeh and Khatchadourian 2019; Yang et al. 2016b). Our final tag list Tfinal contains 64 tags from the top 38 LCSD platforms. We discuss each step in details below.

(1) Identifying Initial low-code tags. The SO posts do not have tags like “low-code” or “lowcode”. Instead, we find that low-code developers use an LCSD platform name as a tag, e.g., “appmaker” for Google Appmaker (2020). Hence, to find relevant tags, first, we compile a list of top LCSD platforms by analyzing a list of platforms that are considered as the market leaders in Gartner (Vincent et al. 2019), Forrester (Rymer et al. 2019), related research work (Sahay et al. 2020), and other online resources like PC magazine (The Best Low-Code Development Platforms 2022). Our compiled list contained 121 LCSD platforms, including all of our previous nine platforms from previous study (Alamin et al. 2021). Then for each of the LCSD platforms, we manually searched for the SO tags in SO. For example, we search for Oracle Apex via SO search engine and find a list of SO posts. We build a potential list of tags related with this platform based on manual inspection, such as “oracle” and “oracle-apex”. Then, manually examine the metadata associated with each of these tags.Footnote 1 For example, “oracle-apex” tag’s metadata says “Oracle Application Express (Oracle APEX) is a rapid Web application development tool that lets you share data and create applications. Using only a Web browser and limited programming experience, you can develop and deploy applications that are fast and secure.” and “oracle” tag’s metadata says “Oracle Database is a Multi-Model Database Management System created by Oracle Corporation. Do NOT use this tag for other products owned by Oracle, such as Java and MySQL.”. Therefore, we select the “oracle-apex” tag for Oracle Apex platform. Not all LCSD platforms have associated SO tags; thus, they were excluded. For example, OneBlink (2022) low-code platform there is no associated SO tags and thus we exclude this from our list. In order to better understand the evolution of this domain, we excluded discontinued LCSD platforms. For example, In Jan 2020, Google announced that they would no longer release new features for Google App Maker and discontinue it by 2021 (2021) and so we excluded this platform from our list. Finally, we found 38 relevant SO tags from 38 platforms. The fifth and the first author participated in this step, and the complete list of the platforms and tags are available in our replication package.

So, our initial list contains 38 LCSD platforms such as: Zoho Creator (2022), Salesforce (2022), Quickbase (2022), Outsystems (2022), Mendix (2022), Vinyl (2022), Appian (2022), and Microsoft Powerapps (2022). We thus focus on the discussions of the above 38 LCSD platforms in SO. We find one tag per LCSD platform as the name of the platform (e.g., “powerapps” for Microsoft Powerapps platform). Thus, We refer to these 38 tags as Tinit.

(2) Finalizing low-code related tags. Intuitively, there might be more variations to tags of 38 LCSD platforms other than those in Tinit. We use heuristics from previous related works (Alamin et al. 2021; Bagherzadeh and Khatchadourian 2019; Yang et al. 2016b) to find other relevant tags. First, we denote our entire SO dump data as Qall. Second, we extract all the questions Q that contain any tag from Tinit. Third, we create a candidate tag list Tcandidate using all the tags found in questions Q. Fourth, we select significantly relevant tags from Tcandidate for our LCSD discussions. Following related works (Uddin et al. 2021b; Bagherzadeh and Khatchadourian 2019; Yang et al. 2016b), we compute significance and relevance for each tag t in Tcandidate with respect to Q (our extracted questions that has Tinit tag) and Qall (i.e., our data dump) as follows,

$$ \begin{array}{@{}rcl@{}} (Significance) \ \ S_{tag} &=& \ \frac{\#\ of\ ques.\ with\ the\ tag\ t\ in\ Q}{\ \ \#\ of\ ques.\ with\ the\ tag\ t\ in\ Q_{all}} \\ (Relevance) \ \ R_{tag} &=& \ \frac{\#\ of\ questions\ with\ tag\ t\ in\ Q}{\ \ \#\ of\ questions\ in\ Q} \end{array} $$

A tag t is significantly relevant to LCSD if the Stag and Rtag are higher than a threshold value. We experimented with a wide range of values of Stag = {0.05, 0.10, 0.15, 0.20, 0.25, 0.30, 0.35} and Rtag = {0.001,0.005, 0.010,0.015,0.020,0.025,0.03}. Figure 2 shows the total number of recommended vs relevant tags from our 49 experiments. It shows that as we increase Stag and Rtag the total number of recommend tags decreases. For example, we find that for Stag = .05 and Rtag = 0.001 the total number of recommended tags is 61 which is highest. However, not all of the suggested tags are LCSD-related. For instance, according to our significance and relevance analysis, tags such as “oracle-xe”, “ems”, “aura-framework”, “power-automate” etc are frequently correlated with other LCSD platform tags, although they do not contain low-code-related discussions. After manually analysing these 61 tags we find that 26 tags are relevant to LCSD-related discussions. So, for the lowest Stag = 0.3 and Rtag = 0.001 we find 26 additional LCSD-related tags. These values are consistent with related work (Uddin et al. 2021b; Alamin et al. 2021; Bagherzadeh and Khatchadourian 2019; Ahmed and Bagherzadeh 2018). The final tag list Tfinal contains 64 significantly relevant tags. So, after combining with out initial taglist, i.e., Tinit, our final tag list Tfinal contains 64 significantly relevant LCSD-related tags which are:

{apex-code, lotus-notes, domino-designer-eclipse, visualforce, salesforce-chatter, apex, salesforce-service-cloud, simple-salesforce, salesforce-ios-sdk, apex-trigger, oracle-apex-5, salesforce-lightning, salesforce-communities, oracle-apex-5.1, servicenow-rest-api, powerapps-formula, salesforce-marketing-cloud, powerapps-selected-items, powerapps-modeldriven, powerapps-collection, powerapps-canvas, oracle-apex-18.2, lwc, salesforce-development, oracle-apex-19.1, oracle-apex-19.2, outsystems, appian, quickbase, powerapps, oracle-apex, salesforce, zoho, mendix, servicenow, pega, retool, vinyl, kissflow, bizagi, neutrinos-platform, rad, joget, filemaker, boomi, opentext, tibco, webmethods, conductor, temenos-quantum, shoutem, oracle-cloud-infrastructure, amazon-honeycode, convertigo, lotus-domino, genero, genesis, gramex, processmaker, orocrm, slingr, unqork, uniface, structr }

Step 3: Extracting low-code related posts. An so question can have at most five tags, and we consider a question as low-code related question if at least one of its tag is in our chosen tag list Tfinal. Based on our Tfinal tag set, we found a total of 27,880 questions from our data dump. SO has a score-based system (up-vote and down-vote) to ensure the questions are in proper language with necessary information (code samples and error messages), not repeated, off-topic or incorrectly tagged. Here is an example for a question with score “-4” where a practitioner is making an API related query in Powerapps (https://stackoverflow.com/questions/61147923/)Footnote 2 platform. However, it is not clear what the practitioner is asking as the question is poorly written and without any clear example. So, in order to ensure good quality discussions, we excluded questions that had a negative score. Following previous research (Uddin et al. 2021b; Bagherzadeh and Khatchadourian 2019; Rosen and Shihab 2016; Barua et al. 2014), we also excluded unaccepted answers and only considered accepted answers for our dataset. Hence, our final dataset B contained 37,766 posts containing 67.4% Questions (i.e., 26,763) and 32.6% Accepted Answers (i.e., 11,010).

Fig. 2
figure 2

Distribution of LCSD related recommended vs relevant tags based on different u and V values from our initial taglist

Full size image

To ensure that our final taglist Tfinal comprises discussions relating to low-code software development, we randomly select 96 questions from our dataset that are statistically significant with a 95% confidence level and 10 confidence interval. First and third authors contributed to this manual analysis, and after manual analysis, we found that 98% of questions from our selected taglist contain low-code platform-related discussion, with only two questions containing discussion that is not particularly related to low-code platforms. For instance, question https://stackoverflow.com/questions/59402662/ includes the tag “appian”, yet the question body describes only about a MySQL database performance-related issue on the Azure platform. Similarly, the question https://stackoverflow.com/questions/19289762/ contains the “apex-code” tag, but exclusively discusses AWS cloud authentication signature-related issues in its problem description.

3.2 Topic Modeling

We produce LCSD topics from our extracted posts in three steps: (1) Preprocess the posts, (2) Find optimal number of topics, and (3) Generate topics. We discuss the steps below.

Step 1. Preprocess the posts. :

For each post text, we remove noise following related works (Abdellatif et al. 2020; Bagherzadeh and Khatchadourian 2019; Barua et al. 2014). First, we remove the code snippets from the body, which is inside <code></code> tag, HTML tags such as (<p></p>, <a></a>, <li></li> etc), and URLs. Then we remove the stop words such as “the”, “is”, “are”, punctuation marks, numbers, non-alphabetical characters using the stop word list from MALLET (McCallum 2002), NLTK (Loper and Bird 2002), and our custom low-code specific (i.e., LCSD platform names) stop word list. We remove the platform’s name from the dataset since, based on our experiments with LDA topic modeling for this study and our past work (Alamin et al. 2021), the resultant topics sometimes tend to cluster around LCSD platforms rather than the technical challenges discussed. Thus, we remove the LCSD platform names from our dataset. After this, we use porter stemmer (Ramasubramanian and Ramya 2013) to get the stemmed representations of the words e.g., “wait”, “waits”, “waiting”, and “waited”—all of which are stemmed to base form “wait”.

Step 2. Finding the optimal number of topics. :

After the prepossessing, we use Latent Dirichlet Allocation (Blei et al. 2003) and the MALLET tool (McCallum 2002) to find out the LCSD-related topics in SO discussions. We follow similar studies in Software engineering research using topic modeling (Arun et al. 2010; Asuncion et al. 2010; Yang et al. 2016b; Bagherzadeh and Khatchadourian 2019; Abdellatif et al. 2020). Our goal is to find the optimal number of topics K for our dataset B so that the coherence score, i.e., encapsulation of underlying topics, is high. We use Gensim package (Rehurek and Sojka 2010) to determine the coherence score following previous works (Uddin and Khomh 2017a; Röder et al. 2015). We experiment with different values of K that range from {5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70} and for each value, we run MALLET LDA on our dataset for 1000 iterations (Bagherzadeh and Khatchadourian 2019). Then we observe how the coherence score is changing with respect to K. We pick the topic model with the highest coherence score. Choosing the right value of K is important because, for smaller values of K, multiple real-world concepts merge, and for a large value of K, a topic breaks down. For example, in our result, the highest coherence score 0.50 for K = 45 and K = 40. The first, third, fourth, and fifth authors participate in the analysis and we choose K = 45 as it captures our underlying topics better. MALLET also uses two hyper-parameters, α and β, to distribute words and posts across the topics. Following the previous works (Alamin et al. 2021; Uddin et al. 2021b; Bagherzadeh and Khatchadourian 2019; Ahmed and Bagherzadeh 2018; Bajaj et al. 2014; Rosen and Shihab 2016), we use the standard values 50/K and 0.01 for hyper-parameters α and β in our experiment.

Step 3. Generating topics. :

Topic modeling is a method of extracting a set of topics by analysing a collection of documents without any predefined taxonomy. Each document has a probability distribution of topics, and every topic has a probability distribution of a set of related words (Barua et al. 2014). We produced 45 topics using the above LDA configuration on our dataset B. Each topic model offers a list of top N words and a list of M posts associated with the topic. In our settings, a topic consists of 30 most frequently co-related words, which represent a concept. Each post had a correlation score between 0 to 1, and following the previous work (Alamin et al. 2021; Uddin et al. 2021b; Wan et al. 2019), we assign a document with a topic that it correlates most.

4 Empirical Study

We report the results of an empirical study by answering the following five research questions (RQ) based on our analysis of LCSD topics in our dataset.

  • RQ1. What topics do LCSD practitioners discuss? (Section 4.1)

  • RQ2. How do the LCSD topics evolve over time in SO? (Section 4.2)

  • RQ3. What types of questions are asked across the observed topic categories? (Section 4.3)

  • RQ4. How are the observed topic categories discussed across SDLC phases? (Section 4.4)

  • RQ5. What LCSD topics are the most difficult to answer? (Section 4.5)

The first two research questions (RQ1, RQ2) provide insights about what topics practitioners discuss in SO and how these topics have evolved over time. The third and fourth research questions (RQ3, RQ4) explore the types of questions in these topics and they affected different SDLC phases. At the end, we discuss the popularity and difficulty of the LCSD topics in the last research question (RQ5).

4.1 What Topics are Discussed About LCSD in Stack Overflow? (RQ1)

4.1.1 Motivation

The increased popularity of LCSD as a flexible and straightforward approach helps develop practical business applications. The challenges of LCSD are yet to be studied as this is a new approach to software development. SO is an established source of knowledge repository to systematically study the real-world challenges that the practitioners face. An understanding of the LCSD topics in SO developer discussions will help LCSD platform providers and researchers to have a better understanding of the underlying prevalent issues, which can then help guide efforts to improve the quality of LCSD platforms.

4.1.2 Approach

We applied LDA topic modeling to our LCSD-related discussion in SO. We get 45 low-code related topics from our LDA topic modeling, as discussed in Section 3. We use card sorting (Fincher and Tenenberg 2005) to label these topics following previous works (Bagherzadeh and Khatchadourian 2019; Ahmed and Bagherzadeh 2018; Yang et al. 2016b; Rosen and Shihab 2016; Abdellatif et al. 2020). In open card sorting, there is no predefined list of labels. Following related works (Alamin et al. 2021; Uddin et al. 2021b; Bagherzadeh and Khatchadourian 2019; Abdellatif et al. 2020), we label each topic by analyzing the top 30 words for the topic and a random sample of at least 20 questions that are assigned to the topic. Four of the authors participated in the labeling process in group sessions (first, third to fifth). Each author assigns a label to each topic and discusses it until there is an agreement. The authors reached an agreement after around 10 iterations of meetings over Skype and email and labeled the 45 topics from the LDA output.

After this initial labeling, we merged a few topics because they contained similar discussions with different vocabularies. For example, we merged topic 36 and 43 into Dynamic form controller because both topics contained discussions related to forms with a predefined list of values, dynamically changing the fields (or options) of forms values based on users’ actions or previous selections. Similarly, we merged topic 2 and 19 to DB Setup & Migration. In the end, we obtained 40 distinct LCSD-related topics.

After the labeling of the topics, we revisited the labels in an attempt to find any clusters/groups among the topics. For example, Date & Time Manipulation, Formatted Data Parsing, and Pattern Matching topics are related, and thus, they are grouped under the General Programming category. We repeated this process multiple times to find increasingly higher categories. For example, we found another category called Dynamic Content which contained two topics Dynamic Data Binding and Dynamic Data Filtering. We then put these two categories under called Business Logic Implementation. This higher abstraction helped us to place other topics related to implementing business logic under this category. Following a similar strategy, we put this Business logic implementation under the Customization category, which discussed customizing applications. For example, under Customization, there were a category called UI which contained Dynamic Layout, and Script category, which contained topics such as Dynamic Page Layout, Dialog Box Manipulation, Window Style Manipulation, and Dynamic Form Controller. The entire process of creating this hierarchy of topic categories took multiple iterations and revisions. We created a coding guideline for creating the taxonomy of topics to ensure consistency and reproducibility. We share the coding guide with our replication package.

4.1.3 Results

After labeling and merging, we find 40 LCSD-related topics. Then after grouping these topics into higher categories, we find five high-level categories: (1) Customization, (2) Data Storage, (3) Platform Adoption, (4) Platform Maintenance, and (5) Third-Party Integration. Figure 3 shows the distribution of topics and questions into these five categories. Among these categories, Customization has the highest coverage of questions and topics (30% Questions in 11 Topics), followed by Data Storage (25% Questions in 9 Topics), Platform Adoption (20% Questions in 9 Topics), Platform Maintenance (14% Questions in 6 Topics), Third-Party Integration (12% Questions in 5 Topics).

Fig. 3
figure 3

Distribution of questions and topics per topic category

Full size image

Figure 4 shows the 40 LCSD topics sorted by numbers of posts. A post means an LCSD-related question or an accepted answer in our case. As discussed in Section 3.1, our dataset has total 37,773 posts containing 26,763 questions and 11,010 accepted answers. The topic with the highest number of posts is placed on top of the list. On average, each topic has 944 posts (question + accepted answer). The topic Window Style Manipulation has the highest number of posts (6.3%) with Questions 5.9% of total questions and 7.2% total accepted answers. On average, each topic has around 669 questions.

Fig. 4
figure 4

Distribution of questions by low-code topics by total number of posts (Number of Questions + Number of Accepted Answers)

Full size image

Figure 5 provides a taxonomy of 40 LCSD related topics into five categories. The topics are organized in descending order of the number of questions. For example, the Customization category has the highest number of questions, followed by Data Storage. Each category may have some sub-categories of topics. For example, the Customization category has two sub-categories: UI and Business Logic. The topics under each sub-category can further be grouped into multiple sub-sub-categories. For example, the UI sub-category has 4 topics grouped into Script and Dynamic Layout sub-sub-categories. Each sub-category, sub-sub-categories, and topics are also presented in descending order of the number of questions.

Fig. 5
figure 5

A taxonomy of LCSD topics with sub-categories

Full size image

We discuss the five categories and the 40 topics in detail below.

Customization Topic Category. :

Customization is the largest topic category in terms of the number of topics and percentage of questions. Out of the 40 topics, 11 topics belong to the Customization category, with around 30% of questions in our dataset. These topics contain discussions about implementing business logic, customizing UI, input and form data validation, general programming-related query to implement some features, etc. This category has two sub-categories: (1)UI contains discussion about customizing the UI, dynamically changing window components, interactive dialog boxes, and (2) Business Logic contains discussion about different programming customization-related queries, dynamically binding UI elements to backend data.

∙ UI Sub-Category contains 15% questions and four topics divided in two sub-sub-categories: (1) Script contains discussion about manipulation of text widgets, formatting components, and (2) Dynamic Layout is about hiding and moving components, showing popups.

The Script sub-sub-category contains 10.4% questions and has two topics: (1) Topic Window Style Manipulation (5.9%) concerns about manipulating the style of the HTML documents such as adding/removing margins/padding (e.g., https://stackoverflow.com/questions/#1/), adding links, manipulating navigation bar and embedded views (e.g., https://stackoverflow.com/questions/#1/). (2) Topic Dynamic Form Controller (4.5%) are about dynamic form, i.e., forms with predefined list of values (e.g., https://stackoverflow.com/questions/#1/), multi select content (e.g., https://stackoverflow.com/questions/#1/), changing forms option based on previous selection (e.g., https://stackoverflow.com/questions/#1/). The Dynamic Layout sub-sub-category covers 4.9% questions and has two topics: (1) Topic Dynamic Page Layout (2.9%) contains discussion about UI (i.e. page) customization (e.g., https://stackoverflow.com/questions/#1/), pagination in https://stackoverflow.com/questions/#1/, hiding or moving element based on some user action or an event (e.g., https://stackoverflow.com/questions/#1/)., (2) Topic Dialog Box Manipulation (2.0%) is about manipulating dialog box (e.g., pop up/ modals) such as hiding them in https://stackoverflow.com/questions/#1/, close them in https://stackoverflow.com/questions/#1/, show popup, refresh web-page (e.g., https://stackoverflow.com/questions/#1/, https://stackoverflow.com/questions/#1/).

∙ Business Logic Sub-Category contains 14.7% questions and 7 topics in two sub-sub-categories: (1) Programming is about discussion related to different programming-related questions and data access, and (2) Dynamic Content is about discussions related to dynamically querying data from different data sources, dynamically changing the web-page content. The Business Logic sub-category contains one topic Conditional BPMN that does not belong to any sub-sub-category. Topic Conditional BPMN (1.6%) contains LCSD platform’s application customization related discussions on Business Process Model and Notation (BPMN) (e.g. https://stackoverflow.com/questions/38265464/) and conditional logic features (e.g., https://stackoverflow.com/questions/66335289/, https://stackoverflow.com/questions/65838553/).

The Dynamic Content sub-sub-category contains 7.6% of questions and has three topics: (1) Topic Dynamic Event Handling (3.1%) discusses about different JavaScript related issues such as JavaScript feature not working (e.g., “JavaScript promise is not working” in https://stackoverflow.com/questions/#1/), browser compatibility, JS event initialization issue (e.g., https://stackoverflow.com/questions/#1/) etc. (2) Topic Dynamic Data Binding (2.6%) is about discussions related to the design of forms with predefined values (e.g., https://stackoverflow.com/questions/#1/), the implementation of multi-select, customized drop-down values, form validation (e.g., https://stackoverflow.com/questions/#1/), changing content of one field based on the value of other field in https://stackoverflow.com/questions/#1/. (3) Topic Dynamic Data Filtering (1.9%) contains business logic customization related discussion based on advanced filtering criteria and querying multiple tables. (e.g., “Find Records Based on the Contents of Records in a Related Table?” in https://stackoverflow.com/questions/#1/ and “find the latest record grouped by name in layout” in https://stackoverflow.com/questions/#1/).

The Programming sub-sub-category contains 5.5% of questions and has three topics: (1) Topic Formatted Data Parsing (2.2%) is about programming related discussion on parsing formatted data, i.e., JSON (e.g., https://stackoverflow.com/questions/#1/, https://stackoverflow.com/questions/#1/), XML (e.g., https://stackoverflow.com/questions/#1/), array of objects (e.g., https://stackoverflow.com/questions/#1/) etc. (2) Topic Pattern Matching (1.8%) topic concerns programming related discussions about searching and modifying strings by pattern matching using regular expression (e.g., “How do I search for an Exact phrase” in https://stackoverflow.com/questions/#1/, “Regex pattern to replace html in a given text string” in https://stackoverflow.com/questions/#1/). (3) Topic Date & Time Manipulation (1.5%) contains programming discussions about date-time manipulation like conversion of formatted string from data-time in https://stackoverflow.com/questions/#1/, calculation of difference between date-time (e.g., https://stackoverflow.com/questions/#1/) and timezone, time conversion (e.g., “How to convert a Date value to unix timestamp?” in https://stackoverflow.com/questions/#1/).

Data Storage Topic Category. :

Data Storage Category is the second largest topic category with a total of 9 topics and around 25% of the questions of our dataset. This topic category contains discussions on database management and file storage. It contains two sub-categories: (1) DBMS is about discussion related to database setup, migration, DB query, (2) File concerns storing and retrieving files (i.e., images, CSV files, etc.).

Platform Adoption Topic Category. :

A total of nine topics belong to the Platform Adoption category with around 20% questions. The nine topics belong to three sub-categories: (1) Documentation contains LCSD platform’s feature-related discussions and how to use those features, (2) Architecture concerns about what type of software development architecture (e.g., client-server communication) is supported by the LCSD platforms, (3) REST API contains LCSD platform’s RESTful APIs.

Platform Maintenance Topic Category. :

We find 6 topics and 13.5% questions in Platform Maintenance category. It has two sub-categories: (1) Configuration contains discussion on LCSD platforms library and build configuration, (2) CI/CD. is about discussion related to DevOps tasks such as continuous integration and continuous delivery, testing etc.

Third-Party Integration Topic Category :

is smallest topic category based on number of questions (12.1% questions). It has five Topics. Four of its topics fall under two sub-categories: (1) REST API contains discussion related to RestFul API communication with third-party services, (2)Plugins is about discussion about external plugins and APIs that are supported by the LCSD platforms.

figure g

4.2 How do the LCSD Topics Evolve over Time? (RQ2)

4.2.1 Motivation

Our analysis of RQ1 finds that LCSD topics are diverse. For example, the Customization topic category contains discussions about developing and customizing the application, and Platform Adoption and Platform Maintenance topic contains discussions related to different features provided by the LCSD platform providers. The platforms for LCSD continue to evolve, as do the underlying topics and question types. We study the evolution of these topics and question types to understand better the evolution and adoption of LCSD development and its community. This analysis will provide valuable insights into the LCSD community and help identify if any topic needs special attention.

4.2.2 Approach

Following related studies (Uddin et al. 2021b), we study the absolute and relative impacts of each of our observed five LCSD topic categories as follows.

Topic Absolute Impact. :

We apply LDA topic for our corpus C and get K topics (t1, t2,........,tk). The absolute impact metric for a topic tk in a month (m) is defined as:

$$ Impact_{absolute}(t_{k};m) = \sum\limits_{p_{i}=1}^{D(m)} \theta (p_{i}; t_{k}) $$
(1)

Here the D(m) is the total number of SO posts in the month m and 𝜃(pi;tk) denotes the possibility for a post (pi) belonging to a topic tk.

From our topic modeling, we found 40 topics that were categorized into five high topic categories, i.e., Customization, Data Storage, Platform Adoption, Platform Maintenance, Third-Party Integration. Now, we further refine the equation for absolute impact for LCSD topics to find absolute impact metrics for a topic category (TCj) for a month m as follows:

$$ Impact_{absolute}(TC_{j};m) = \sum\limits_{t_{k}}^{TC_{j}}Impact_{absolute}(t_{k}; m), 0 < j < TC $$
(2)
Topic Relative Impact. :

Related impact metric signifies the proportion of posts for a particular LCSD topic tk relative to all the posts in our corpus C for a particular month m. Following related studies (Uddin et al. 2021b), we compute the related impact metrics for LCSD topics. We use the following equation to compute the metric for a topic tk in a month m as follows:

$$ Impact_{relative}(t_{k},m) = \frac{1}{|D(m)|}\sum\limits_{p_{i}=1}^{\theta}(p_{i}; t_{k}), 1\le i\le C $$
(3)

Here D(m) denotes the total number of posts that belongs to a topic tk for a particular month m. Here δ denotes the probability of a particular post pi for our Corpus C belonging to a particular topic tk.

Similar to the absolute impact, we refine the equation to compute the relative impact on LCSD topic categories as follows:

$$ Impact_{relative}(TC;m) = \sum\limits_{t_{k}}^{TC}Impact_{relative}(t_{k};m) $$
(4)

Here TC donates one of our five topic categories and topics that belong to each topic category.

4.2.3 Result

Figure 6 depicts the progression of overall LCSD-related conversation using the absolution impact equation from our extracted dataset between 2008 to 2021. Additionally, it demonstrates that the peaks of LCSD-related talks occurred in mid-2020 (i.e., around 400 questions per month). It also reveals that LCSD-related discussion has been gaining popularity since 2012. In the section below, we provide a more extensive explanation for the minor spikes in the Fig. 6.

Fig. 6
figure 6

The evolution of overall LCSD-related discussions over time

Full size image

We observe that in the early days (i.e., 2009), Platform Adoption along with Data Storage topic category has more questions compared to Customization. Customization topic category starts to get a dominant position from mid (i.e., August) of 2011 over Platform Adoption, and it remains the dominant topic till the end of 2021. The number of questions in the Customization topic category gradually increased over time, from August 2011 (23) to March 2012 (81) to May 2020 (99). Data Storage topic category briefly exceeds Customization Category during August 2013, but it mostly holds a dominant position other times compared to Platform Adoption topic category. On the other hand, Platform Maintenance and Third-Party Integration exhibits very similar evolution over the whole time.

We further look into the Fig. 7 and see there are mainly two significant upticks in the number of posts about LCSD. The first one is between August 2011 to May 2012, when there is a sharp increase in the number of questions for almost all topic categories, especially for Customization and Data Storage Category. By this Salesforce (2022) LCSD platform-related discussion becomes quite popular in SO, and around that time, it ranks very high as a CRM platform. The second increase in posts is between February 2020 and August 2020. During this time of the Covid19 pandemic, many businesses began to operate online and there is a significant uptick in the number of questions in the Customization category, followed by Data Storage and Platform Adoption Moreover, there is an uptick in the number of questions on building a simple app to collect and report data, especially the Salesforce platform. There is an increase in the Platform Adoption topic category between mid-2016 to mid-2017. During this time Oracle Apex platform released version 5.0, and there is an uptick of questions regarding different features such as interactive grid in https://stackoverflow.com/questions/43316233/, drag and drop layout in https://stackoverflow.com/questions/45818292/. Now we provide a detailed analysis of each of the five topic categories.

Fig. 7
figure 7

The absolute impact for LCSD topic categories over time

Full size image
Customization :

This is the biggest topic category with 11 topics. From 2008 to mid of 2011, all of these topics evolve homogeneously. From the mid of 2011 to the first Quartile of 2012, Dynamic Page Layout topic becomes dominant. “How to get fields in page layout” in https://stackoverflow.com/questions/7256190/, issues with page layout in different LCSD platforms (e.g., https://stackoverflow.com/questions/7421985/). From the end of 2012 to 2017, Window Style Manipulation topic remains most dominant. “Passing values from child window to parent window which is on another domain?” in https://stackoverflow.com/questions/16463602/, view related issues https://stackoverflow.com/questions/15715645/. From the end of 2017 to the end, our dataset Dynamic Form Controller topic remains the most dominant.

Data Storage Category :

From mid-2015, Database Setup & Migration topic becomes the most dominant topic in this category and has some high spikes during the pandemic and mid of 2017. For instance, there are queries like “Using Jenkins for OCI database migration” in https://stackoverflow.com/questions/62217796/ and “Almost all the cloud service providers have 99.95% of data available on the cloud. What will happen if the whole region sinks in an earthquake?” in https://stackoverflow.com/questions/62102679/. Since 2017 DevOps and database “Domino Xpage database building automation or continuous integration using Jenkins with maven.” in https://stackoverflow.com/questions/43092239/. From mid-2011 to mid-2014, DB Stored Procedure topic remains dominant. “Oracle APEX: Call stored procedure from javascript” in https://stackoverflow.com/questions/20501834/.

Platform Adoption Category :

From 2008 to mid-2011, Platform Adoption related topics were the most dominant (e.g., “Suggested platform/tools for rapid game development and game prototyping” in https://stackoverflow.com/questions/312357/). Between mid-2011 to mid-2017, Authentication & Authorization topic becomes dominant (e.g., “Can I implement my own authentication process in force.com or it is against terms of service?” https://stackoverflow.com/questions/13059568/, https://stackoverflow.com/questions/13034866/). Since the end of 2017, Platform Infrastructure API remains the most dominant. So, practitioners ask queries like “VirtualBox VM changes MAC address after imported to Oracle Cloud Infrastructure” in https://stackoverflow.com/questions/61501108/ and “How to send a classic report as mail body in oracle Apex 19.2” in https://stackoverflow.com/questions/59693984/, report layout (https://stackoverflow.com/questions/59833909/, https://stackoverflow.com/questions/59752159/).

Platform Maintenance Topic Category :

From 2008 to mid-2019, the Build Configuration Management topic remains the most dominant topic. It has some high spikes in the number of questions during the beginning of 2012 and the first quartile of 2014. Build error https://stackoverflow.com/questions/21720165/, https://stackoverflow.com/questions/21326163/, build projects automatically https://stackoverflow.com/questions/21758244/. From mid-2019, Library Dependency Management topic-related questions became popular (e.g., library-related issues (e.g., https://stackoverflow.com/questions/62825046/, https://stackoverflow.com/questions/61100705/), library not foundhttps://stackoverflow.com/questions/61911916/).

Third-Party Integration Topic Category. :

The five topics from this category evolve simultaneously. From the beginning of 2015, the External Web Request Processing topic has become more dominant than other topics with a slight margin. External Web Request Processing and Fetch & Process API response, E-signature topics become dominant during the pandemic with queries such as platform support on e-signature https://stackoverflow.com/questions/62417381/ and etc.

In Fig. 8, we now provide more insight into the evolution of LCSD topic categories. It confirms the findings presented in Fig. 7 and adds some previously unknown insights. For instance, in the last quartile of 2009, it is apparent that Data Storage is the most popular Topic Category. According to the absolute impact metric, all five themes are increasing monotonically. The relative impact measure, on the other hand, indicates that the Customization, Platform Maintenance, and Third-Party Integration Topic group evolves in a nearly identical manner. However, this Figure demonstrates that, beginning in 2016, Platform Adoption-related conversation increased and eventually surpassed Data Storage-related discussion. This in-depth examination of evolution is significant because it demonstrates that, while Data Storage Topics are the second-largest Topic category, Platform Adoption-related queries are evolving rapidly and require further attention from platform vendors.

figure h
Fig. 8
figure 8

The relative impact for LCSD topic categories over time

Full size image

4.3 What Types of Questions are Asked Across the Observed Topic Categories? (RQ3)

4.3.1 Motivation

This research question aims to provide a deeper understanding of LCSD-related topics based on the types of questions asked about the LCSD platforms in SO. For example, “what” types of questions denote that developers are not sure about some specific characteristics of LCSD platforms, while “how” types of questions denote that they do not know how to solve a problem using an LCSD platform. Intuitively, the prevalence of “what” types of questions would denote that the LCSD platforms need to better inform the services they offer, while the prevalence of “how” type of questions would denote that the LCSD platforms need to have better documentation so that developers can learn easily on how to use those. Initially, in 2011 Treude et al. (2011) investigated different types of questions on stack overflow. Later Rosen and Shihab (2015) conducts an empirical study like ours on Mobile developers’ discussion in stack overflow with these four types of questions. Later, very similar studies on chatbot development (Abdellatif et al. 2020) and IoT developers’ discussion on Stack overflow (Uddin et al. 2021b) also explore this research question to provide more insights about specific domains and complement the findings of topic modeling.

4.3.2 Approach

In order to understand what-type of questions are discussed in SO by the LCSD practitioners, we take a statistically significant sample from our extracted dataset and then manually analyze each question and label them into one of four types: How-type, Why-type, What-type, Others-type following related studies (Uddin et al. 2021b; Abdellatif et al. 2020; Rosen and Shihab 2015). So, our approach is divided into two steps: Step 1. We generate a statistically significant sample size, Step 2. we manually analyze and label them.

Step 1. Generate Sample. :

As discussed in Section 3.1 our final dataset has 26,763 questions. A statistically significant sample with a 95% confidence level and five confidence intervals would be at least 379 random questions, and a 10 confidence interval would have a sample size of 96 questions. A random sample represents a representative for the entire dataset, and thus this could miss questions from the subset of questions that may belong to smaller topic categories. For example, as discussed in RQ1, we have 40 topics organized into five categories. As random sampling is not uniform across the topic categories, it might miss important questions from smaller topic categories such as Third-Party Integration. Therefore, following previous empirical studies (Uddin et al. 2021b; Abdellatif et al. 2020), we draw a statistically significant random sample from each of the five topic categories. Specifically, we draw the distribution of questions in our sample from each of the 5 topic categories with a 95% confidence level and ten confidence intervals. The sample is drawn as follows: 95 questions from the Customization category (total question 8014), 95 questions from the Data Storage category (total question 6610), 94 questions from Platform Adoption category (total question 5285), 94 questions from Platform Maintenance category (total question 3607), 93 questions from Third-Party Integration category (total question 3247). In summary, we sampled a total of 471 questions.

Step 2. Label Question Types. :

We analyze and label each question from our samples into the following four categories. The categories and the coding guides follow previous research (Uddin et al. 2021b; Abdellatif et al. 2020; Rosen and Shihab 2016)

  • How-type post contains a discussion about the implementation details of a technical task (Uddin et al. 2021b). The questions primarily focus on the steps required to solve certain issues or complete certain tasks (e.g., “How to create a submit button template in Oracle APEX?” in https://stackoverflow.com/questions/#1/).

  • Why-type post is about troubleshooting and attempting to determine the cause/reason for a behavior. These questions help practitioners understand the problem-solving or debugging approach, e.g., in https://stackoverflow.com/questions/#1/, a user is trying to find out why an SSL server certificate is rejected.

  • What-type question asks for more information about a particular architecture/event. The practitioners ask for more information that helps them to make informed decisions. For example, in https://stackoverflow.com/questions/#1/ a practitioner is asking for detailed information about the Oracle Apex platform’s secure cookies.

  • Other-type question do not fall into any of the above three categories, e.g., “Initiating Salesforce API in Google App Script” in https://stackoverflow.com/questions/#1/

Three authors (first, third and fourth) participated together in the labeling process. We assessed our level of agreement using Cohen kappa(McHugh 2012). The disagreement and annotation difficulties were resolved by discussing with the first author. In general, the authors achieved a substantial agreement (k > 0.6) on the 471 questions classified. Our coding guidelines and the final annotated dataset are available in our replication package.

4.3.3 Result

Table 1 shows the percentage of type of questions across our five LCSD topic categories. Similar to related studies (Uddin et al. 2021b; Abdellatif et al. 2020; Rosen and Shihab 2016), during our labeling process, we observed that some of the questions can have multiple labels, e.g., What-type and How-type. For example, “How can I get jQuery post to work with a [platform] WebToLead” in https://stackoverflow.com/questions/2339550/ discusses making Ajax request using jQuery where the practitioner is getting an error response. At the same time, the practitioner is further querying some detailed information on how Ajax requests. Therefore, the sum of percentages of question types reported in the result section is more than 471 classified posts. We now discuss each question type with examples (Fig. 9).

How-type. :

Around 57% of our annotated questions fall under this type. This type of question is most prevalent in the topic categories Third-Party Integration (61%) followed by Data Storage (60%), Platform Adoption (57%), Customization (51%), Platform Maintenance (49%). This high prevalence is not surprising, given that SO is a Q&A platform and the LCSD practitioners ask many questions about how to implement certain features or debug an issue. Additionally this also signifies that LCSD practitioners are asking a lot questions while integrating with third-party library (e.g., https://stackoverflow.com/questions/62825046/) and plugins (e.g., https://stackoverflow.com/questions/61455233/) and managing the data with a database management system (e.g., https://stackoverflow.com/questions/38111768/) or file storage (e.g., https://stackoverflow.com/questions/63284305/). To explain further we find questions regarding implementing encryption, e.g., https://stackoverflow.com/questions/2220076/, or Making HTTP POST request, e.g., https://stackoverflow.com/questions/32736416/, or debugging a script, e.g., https://stackoverflow.com/questions/45619586/, or implement a feature, e.g., https://stackoverflow.com/questions/28990848/ etc.

What-type. :

This is the second biggest question type with 18% of all annotated questions. This type of question is the most dominant in the topic categories Platform Maintenance (20.5%), Platform Adoption (20%), and Customization (18%). This type of question can be associated with How-type questions, where the practitioners require further information to implement certain features. For instance, in this question, a practitioner is querying about “How to implement circular cascade select-lists” in https://stackoverflow.com/questions/60676786/. The questions in this category signify that practitioners fail to find relevant information from the official documentation (e.g., https://stackoverflow.com/questions/9377042/) sources. Therefore, as this type of question is prevalent in Platform Maintenance and Platform Adoption category, LCSD platform providers might focus more on improving their resources. As an example, we find questions on JavaScript events not working correctly, e.g., https://stackoverflow.com/questions/51564154/, roll back changes, e.g., https://stackoverflow.com/questions/11156810/, designing workflow, e.g., https://stackoverflow.com/questions/11156810/.

Why-type. :

This is the third most prevalent question type category, with 14% of all annotated questions. This type of question is the most prevalent in the topic categories Customization, Data Storage, and Platform Maintenance with around 17% questions. These questions are mostly related to troubleshooting like when LCSD practitioners implement particular features or deploy an application. For instance, e.g., “Why does this error happen in [Platform]?” in https://stackoverflow.com/questions/48818859/, “Why isn’t the document going into edit mode” in https://stackoverflow.com/questions/51660117/, “Not able to launch Android Hybrid app using [Platform] Mobile SDK” in https://stackoverflow.com/questions/20417235/, “Java code running twice” in https://stackoverflow.com/questions/17147921/.

Other-type. :

Around 14% of our annotated questions fall under this type. The questions are almost uniformly distributed across the five topic categories. The questions contain general problems, e.g., “UTF-8 character in attachment name” in https://stackoverflow.com/questions/22808965/ or “Domino Server is installed on Unix or Windows?” in https://stackoverflow.com/questions/10796638/. Some of the questions in this type also contain multiple/ambiguous questions (e.g., https://stackoverflow.com/questions/27896327/). For example, How to test an application?, which library is better? etc.

figure i
Table 1 Types of questions across the LCSD topic categories
Full size table
Fig. 9
figure 9

Distribution of different question Types

Full size image

4.4 How are the Observed Topic Categories Discussed Across SDLC Phases? (RQ4)

4.4.1 Motivation

As we observed the prevalence and evolution of diverse LCSD topics in SO, we also find that the topics contain different types of questions. This diversity may indicate that the topics correspond to the different SDLC phases that are used to develop low code software development (see Section 2 for an overview of the LCSD phases). For example, intuitively What-type of questions may be due to the clarification of a low code software requirements during its design phases, which questions/topics related to troubleshooting of issues may be asked during the development, deployment, and maintenance phase. Therefore, an understanding of the SDLC phases in the LCSD questions in SO may offer us an idea about the prevalence of those SDLC phases across our observed LCSD topics in SO. This understanding may help the LCSD practitioners to determine how SO can be used during low code software development across the various SDLC phases.

4.4.2 Approach

In order to understand the distribution of LCSD topics across agile SDLC phases, we collect a statistically significant sample of questions from our extracted dataset D into one of the six Agile software development methodology (Beck et al. 2001) phases: (1) Requirement Analysis & Planning, (2) Application Design, (3) Implementation, (4) Testing, (5) Deployment, and (6) Maintenance. First, we generate a statistically significant sample size. We use the same set of randomly selected (i.e., 471) posts that we produced during RQ3 (see Section 4.3). So, we take a statistically significant stratified random sample for each topic category in our dataset with 95% confidence level and 10 confidence interval to ensure that we have a representative sample from each topic category (Uddin et al. 2021b; Abdellatif et al. 2020). We manually annotate each question post with one/more SDLC phases.

We followed the same annotation strategy to label SDLC phased as we did for RQ3 (see Section 4.3.2). Each question was labeled by at least two authors (second and third/fourth) after extensive group discussion on formalizing annotation guidelines and participating in joint sessions. We find our level of agreement using Cohen kappa (McHugh 2012; Alamin et al. 2021). The authors generally achieved a substantial agreement (k > 0.70). For example, a new practitioner is tasked with finding the right LCSD platform during the planning stage of his/her LCSD application. The practitioner queries, “Are there any serious pitfalls to Outsystems Agile Platform?” (https://stackoverflow.com/questions/3016015/). We thus assign the SDLC phase as “Requirement Analysis & Planning”. Another question asks, “Google App Maker app not working after deploy” (https://stackoverflow.com/questions/42506938/). We label the SDLC phase as “Deployment”. For some questions, it involved significant manual assessment to assign appropriate SDLC phase, e.g., Requirement Analysis & Planning phase vs Application Design and Application Design vs Implementation phase. As such, we developed a detailed annotation/coding guide to help us with the manual assessment. This annotation guide was constantly updated during our study to ensure that the guide remained useful with all relevant instructions. For example, one of the questions that helped refine our annotation guide is the question noted by the respected reviewer, i.e., “Can AppMaker be utilized with SQL Server?” in https://stackoverflow.com/questions/55220499/. The user in this question wants to know if Google App Maker and SQL Server databases can be connected. This question was categorized as Application design phase. Based on this, according to our annotation guideline, this question can be labelled as Requirement Analysis & Planning phase too. However, after discussion, the first and third authors agreed to label it as Application Design phase because from the problem description, it seems the question mainly focuses on connecting the application to a custom data storage. As this question focuses on data source design, which is often explored during the Application Design phase, we concluded that it should be labeled as such. The labeling of each question to determine the precise SDLC phases was conducted by several co-authors in joint discussion sessions spanning over 80 person-hours.

4.4.3 Results

Figure 10 shows the distribution of our LCSD questions into six agile SDLC phase. We find that the Implementation phase has 65% of our 471 annotated questions, followed by Application Design (17%), Requirement Analysis & Planning (9.1%). It is not surprising that the Implementation phase has so many questions because SO is a technical Q&A platform and practitioners use it mostly to find issues when trying to implement some feature. Though the percentage of questions is not very high (e.g., between 2–3%), we also find practitioners ask questions regarding Testing and Deployment phases too (e.g., “Automated Testing for Oracle [Platform] Web Application” in https://stackoverflow.com/questions/1764497/). This analysis highlights that LCSD practitioners ask questions regarding the feasibility analysis of a feature to make design decisions to implement the feature to deployment. We provide an overview of the types of questions asked during these six SDLC phases.

Fig. 10
figure 10

Distribution of questions (Q) per SDLC phase

Full size image

Requirement Analysis & Planning (43, 9.1%) Requirement analysis is the first and most important stage of software development because the application largely depends on this. Requirement analysis is a process to develop software according to the users’ needs. In agile software development methodology, features are implemented incrementally, and requirement and feasibility analysis are crucial in implementing a new feature. During this phase, the operational factors are considered, and the feasibility, time-frame, potential complexity, and reliability. Requirement management tools are typically included with LCSD systems, allowing developers to collect data, modify checklists, and import user stories into sprint plans. Throughout this stage, developers tend to ask questions regarding the platform’s features (e.g., “Does Mendix generates a source code in any particular language, which can be edited and reused?” in https://stackoverflow.com/questions/53043346/), learning curve (e.g., https://stackoverflow.com/questions/55304547/, https://stackoverflow.com/questions/45631057/), and the LCSD platform’s support for faster application development (e.g., https://stackoverflow.com/questions/28983651/), general deployment/maintenance support (e.g., https://stackoverflow.com/questions/50460088/) in order to select the best platform for their needs. For example, in this popular question, a new practitioner is asking for some drawbacks on some potential pitfalls for a particular LCSD platform, e.g., “Are there any serious pitfalls to [Platform] Agile Platform?” (https://stackoverflow.com/questions/3016015/). A developer from that platform provider suggests using the platform to build an application and decide for himself as it is hard to define what someone might consider a pitfall. In another question, a practitioner is asking if it is possible to integrate Selenium with an LCSD platform (e.g., https://stackoverflow.com/questions/52010004/)

Application Design (80, 17%) The design specification is created in this step based on the application’s needs. The application architecture (e.g., https://stackoverflow.com/questions/53820097/), modularity, and extensibility are all reviewed and approved by all critical stakeholders. The LCSD developers face challenges regarding data storage design, drag and drop UI design, connecting on-premise data-sources with the LCSD platform (e.g., “Can AppMaker be used with SQL Server” (https://stackoverflow.com/questions/55220499/)), data migration to LCSD platform (https://stackoverflow.com/questions/46421271/), following best practices (e.g, “Salesforce Best Practice To Minimize Data Storage Size” in https://stackoverflow.com/questions/14073151/), designing a responsive web page (e.g., (https://stackoverflow.com/questions/52744026/)).

Implementation (306, 65%) The actual application development begins at this phase. LCSD developers confront a variety of obstacles when they try to customize the application (i.e., personalize UI (e.g, https://stackoverflow.com/questions/6454308/), implement business logic (e.g, https://stackoverflow.com/questions/40472354/)), integrate third-party plugins(e.g, https://stackoverflow.com/questions/46538734/), debug (e.g, https://stackoverflow.com/questions/35898112/) and test the implemented functionality. For example, LCSD practitioners ask customization questions such as How can they change the timezone in a platform in https://stackoverflow.com/questions/47731051/, customizing UI in https://stackoverflow.com/questions/40159662/. Many of these challenges arise from incomplete or incorrect documentation. In https://stackoverflow.com/questions/34510911/, an LCSD developer asks for sample code to convert a web page to a PDF. The official documentation is not sufficient enough for entry-level practitioners.

Testing (13, 2.7%) LCSD testing differs from standard software testing in some fundamental ways. In LCSD development, many of the features are implemented using graphical interfaces, and they are provided and tested by the LCSD platform providers. As a result, unit testing is less important compared to traditional software development. In LCSD approach practitioners face difficulties to lack of documentation of testing approach in LCSD platform (e.g, “How to bypass login for unit-testing [Platform]?” in https://stackoverflow.com/questions/54432666/), test coverage (e.g, https://stackoverflow.com/questions/54899980/, https://stackoverflow.com/questions/57755398/), automated testing (e.g, “[Platform] 20.1 automated testing” https://stackoverflow.com/questions/63594106/), testing browser compatibility (e.g, https://stackoverflow.com/questions//), troubleshooting errors while running tests (e.g, https://stackoverflow.com/questions/47254010/) etc.

Deployment (16, 3.3%) At this phase, the feature of the application needs to be deployed for the targeted users. One of the goals of LCSD development is to handle many of the complexities of the deployment and maintenance phase. Many LCSD platform providers provide advanced Application Life-Cycle Management tools to deploy and maintain the staging (i.e., testing) and the production server (e.g., https://stackoverflow.com/questions/65124133/). However, LCSD practitioners still face many challenges regarding deployment configuration issues (https://stackoverflow.com/questions/46369742/), Domain name configuration (e.g., DNS configuration (e.g, https://stackoverflow.com/questions/65678735/), SSL Configuration (e.g, https://stackoverflow.com/questions/67186273/)), accessibility issues such as with public URL (https://stackoverflow.com/questions/44136328/, https://stackoverflow.com/questions/53884162/)) etc. For example, in this post, a practitioner is having deployment issues (e.g., “[Platform] app not working after deployment” (https://stackoverflow.com/questions/42506938/)). A community member provides a detailed description of how to accomplish this in the answer, highlighting the lack of Official Documentation for such a critical use-case. There are a few questions concerning delivering an app with a custom URL or domain name (for example, “How to make friendly custom URL for deployed app” in https://stackoverflow.com/questions/47194231/). It was challenging in this scenario because the platform did not have native support.

Maintenance (13, 2.8%) At this phase, the LCSD application is deployed and requires ongoing maintenance. Sometimes new software development life cycle is agile (i.e., incremental) because new issues are reported that were previously undiscovered and request new features from the users. LCSD practitioners face some problems at this phase, such as event monitoring (e.g, https://stackoverflow.com/questions/64322219/), collaboration and developers role management (e.g, “Role based hierarchy in report access” in https://stackoverflow.com/questions/10436719/ or https://stackoverflow.com/questions/52762374/), and application reuse (e.g, https://stackoverflow.com/questions/64276891/), application version, i.e., “Do I have the latest version of an [Platform] component?” in https://stackoverflow.com/questions/45209796/ or https://stackoverflow.com/questions/52762374/, etc.

Table 2 Distribution (frequency) of LCSD topics per SDLC phase. Each colored bar denotes a phase (Black = Requirement Analysis, Green = Application Design, Magenta = Implementation, Red = Testing, Blue = Deployment, Orange = Maintenance)
Full size table
Topic Categories in different SDLC phases :

We find that for all five topic categories, LCSD practitioners need some community support from planning to debugging to deployment (e.g., “How does one deploy after building on [platform]” in https://stackoverflow.com/questions/3952481/). We report how LCSD topics and different types of questions are distributed across six SDLC phases. Table 2 shows the distribution of SDLC phases for each topic category. Our analysis shows that for the Customization topic Category, most questions are asked during the Implementation (75%) and Design (18%) phases. The most dominant SDLC phase, i.e., the Implementation phase, is most prevalent in Customization (75%), Data Storage (74%), and Third-Party Integration (73%). Requirement Analysis phase is dominant in Platform Adoption (18%) and Platform Maintenance (12%) topic categories where practitioners ask questions like “Disadvantages of the [platform]” in https://stackoverflow.com/questions/1664503/. Similarly, question in Platform Maintenance topic category is also prevalent in Testing (11%), deployment (13%), and Maintenance (6%) SDLC stage.

Types of questions in different SDLC phases :

We report the distribution of question types across SDLC phases in Table 3. It shows that for Requirement Analysis & Planning phase, most questions (35%) belong to What-type. This insight signifies that at this phase, practitioners are making inquiries about feature details (e.g., https://stackoverflow.com/questions/9577099/). In the Application Design, Implementation, and testing phase, most of the questions belong to How-type, i.e., practitioners are querying about how they can implement a particular feature (e.g., https://stackoverflow.com/questions/13933003/) or test it (e.g., https://stackoverflow.com/questions/9594709/). At the Deployment phase most prominent is Why-type (38%) followed by How-type(31%). We can see a similar pattern for the Maintenance phase, where the most significant question type is How-type (46%) followed by Why-type (31%). We see this pattern because, at the Deployment and Maintenance phase, most of the questions belong to some server configuration error (e.g., https://stackoverflow.com/questions/4497228/) and the practitioners’ inquiry about how they can set up specific server settings (e.g., https://stackoverflow.com/questions/8148247/). Similarly, we find that What-type questions are more prevalent during Requirement Analysis and Deployment phases.

figure k
Table 3 Types of questions across the Software development life cycle phases
Full size table

4.5 What LCSD Topics are the Most Difficult to Get an Accepted Answer? (RQ5)

4.5.1 Motivation

After reviewing LCSD-related topics and discussions in the agile SDLC stages, we discovered that LCSD practitioners encounter generic software development problems and particular challenges specific to LCSD platforms (e.g., Platform Adoption, Platform Maintenance). Some posts come up repeatedly, and some have a lot of community participation (i.e., answers, comments, up-votes). As a result, not all topics and SDLC phases are equally difficult to get a solution. A thorough examination of the complexity and popularity of the practitioners’ conversation might yield valuable information about how to prioritize research and community support. For example, LCSD platform providers and academics can take the required measures to make the architecture, design, features, and tools of LCSD platforms more useable for practitioners, particularly newbies.

4.5.2 Approach

We compute the difficulty of getting an accepted answer for a group of questions using two metrics for each question in that group (1) Percentage of questions without an accepted answer, (2) Average median time needed to get an accepted answer. In the same way, we use the following three popularity metrics to calculate popularity of that topic in the SO community: (1) Average number of views, (2) Average number of favorites (i.e., for each question number of users marked as favorite), (3) Average score.

The five metrics are standard features of a SO question, and many other related studies (Uddin et al. 2021b; Alamin et al. 2021; Bagherzadeh and Khatchadourian 2019; Abdellatif et al. 2020; Ahmed and Bagherzadeh 2018) have used them to analyze the popularity and difficulty of getting a solution for a question. In SO, one question can have multiple answers, and The user who posted the question has the option of marking it as accepted. Hence, the accepted answer is considered correct or sound quality. So, the absence of an accepted answer may indicate the user did not find a helpful, appropriate answer. The quality of the question (i.e., problem description) might be one reason for not getting an acceptable answer. However, the SO community collaboratively edits and improves the posts. Therefore, the lack of an accepted answer most likely indicates that the SO community finds those questions challenging to answer. The success and usefulness of a crowd-sourced platform such as SO depends on the community members to quickly provide relevant, helpful correct information. In SO, the median time to get an answer is around 21 min only (Uddin et al. 2021b), but a complicated or domain-specific question may necessitate additional time to receive an accepted answer.

It can be non-trivial to assess the popularity and difficulty of getting an accepted answer for the topics using multiple metrics. We thus compute two fused metrics following related works (Uddin et al. 2021b). We describe the two fused metrics below.

Fused Popularity Metrics First, we compute the popularity metrics for each of the 40 LCSD topics. However, the average view counts can be in the range of hundreds, average scores, and average favorite count between 0-3. Therefore, following related study (Uddin et al. 2021b) we normalize the values of the metrics by dividing the metrics by the average of the metric values of all the groups (e.g., for topics K = 40). Thus, we create three new normalized popularity metrics for each topic. For example the normalized metrics for a group i for all the K groups can be V iewNi, FavoriteNi, ScoreNi (e.g., for LCSD topics K = 40). Finally, We calculate the fused popularity FusedPi of a group i by taking the average of the three normalized metric values.

$$ \begin{array}{@{}rcl@{}} ViewN_{i} = \frac{View_{i}}{\frac{{\sum}_{j=1}^{K}View_{j}} {K}} \\ FavoriteN_{i} = \frac{Favorite_{i}}{\frac{{\sum}_{j=1}^{K}Favorite_{j}} {K}} \\ ScoreN_{i} = \frac{Score_{i}}{\frac{{\sum}_{j=1}^{K}Score_{j}} {K}} \end{array} $$
(5) (6) (7)
$$ FusedP_{i} = \frac{ViewN_{i} + FavoriteN_{i} + ScoreN_{i}}{3} $$
(8)

Fused Difficulty Metrics Similar to popularity metrics, we first compute the difficulty metrics for each topic. Then we normalize the metric values by dividing them by the average of the metric value across all groups (e.g., 40 for LCSD topics). Thus we, create two new normalized metrics for a given topic i. Finally, We calculate the fused difficulty metric FusedDi of topic i by taking the average of the normalized metric values.

$$ \begin{array}{@{}rcl@{}} PctQuesWOAccAnsN_{i} = \frac{PctQWoAcceptedAnswer_{i}}{\frac{{\sum}_{j=1}^{K}PctQWoAcceptedAnswer_{j}}{K}} \\ MedHrsToGetAccAnsN_{i} = \frac{MedHrsToGetAccAns_{i}}{\frac{{\sum}_{j=1}^{K}MedHrsToGetAccAns_{j}}{K}} \end{array} $$
(9) (10)
$$ FusedD_{i} = \frac{PctQuesWOAccAnsN_{i} + MedHrsToGetAccAnsN_{i}}{2} $$
(11)

In addition to this, we also aim to determine the correlation between the difficulty and the popularity of the topics. We use the Kendall Tau correlation measure (Kendall 1938) to find the correlation between topic popularity and topic difficulty. Unlike Mann-Whitney correlation (Kruskal 1957), it is not susceptible to outliers in the data. We can not provide the evolution of popularity and difficulty for these topics because SO does not provide the data across a time series for all metrics such as view count, score, etc. However, asLCSD-related topics are showing increasing trends in recent times, our analysis is valid for recent times.

4.5.3 Results

In Fig. 11 we present an overview of the five high-level topic categories and their popularity and difficulty to get an accepted answer. In the Figure, the bubble size represents the number of questions in that category. The Figure shows that Platform Adoption is the most popular and challenging topic category to get an accepted answer, followed by Customization, Data Storage, Platform Maintenance, and Third-Party Integration. We can also see that three topic categories, Platform Maintenance, Data Storage, and Customization, are almost similar in terms of difficulty to get a solution. From our analysis, we find that practitioners find the Third-Party Integration topic category relatively less difficult because many questions in this category are also relevant to traditional software development (e.g., integrating Google Maps in https://stackoverflow.com/questions/63457325/ and https://stackoverflow.com/questions/1258834/) and thus easier to get community support. Similarly, we find that questions in the Platform Adoption topic category are quite specific to particular LCSD platforms and thus sometimes have less community support to find an acceptable answer quickly.

Fig. 11
figure 11

The popularity vs. difficulty of getting an accepted answer for LCSD Topic categories

Full size image

Topic Popularity For each of the 40 topics, Table 4 shows three popularity metrics: Average number of 1. Views, 2. Favorites, 3. Scores. It also contains the combined popularity metrics (i.e., FusedP) that are based on the above three metrics and using the (8). In the Table, the topics are presented in descending order based on the FusedP popularity metric.

Table 4 Popularity for getting an accepted answer for LCSD topics
Full size table

Platform Related Query topic from the Platform Adoption Category has the highest FusedP score. It also has the highest average favorite count (e.g., 0.90) and highest average score (e.g., 2.60). 1.7% of total questions. This topic contains discussion about LCSD platforms features of different platforms, software development methodologies such as Agile and RAD development. The topic Message Queue under Platform Adoption category has the second highest FusedP value. This topic is about different asynchronous service-to-service data exchange mechanisms such as using a message queue. It generally contains discussions about popular micro-service design patterns. The topic Dynamic Page Layout under Customization categories is the third most popular topic and it has the highest average view count (e.g., 2447.2). The posts under this topic discuss about UI (i.e. page) customization, hiding or moving elements based on some user action or an event (e.g., disable a button for dynamic action in https://stackoverflow.com/questions/8640964/. The eSignature topic from Third-Party Integration is the least popular with only 1.15% of total questions, a fused value of 0.52. It has the lowest favorite and score count. This contains discussion about different issues and customization for electronic signature of documents, i.e., docusign about collecting user’s agreement/permission for sales or account opening. This topic is not that much popular and easy to get an accepted answer because this requirement is not generalized and not all the low-code application requires this.

Topic Difficulty In Table 5 we present the two difficulty metrics: for all the questions in a topic 1. Percentage of questions without accepted answers, 2. Median hours to get accepted answer. Similar to topic popularity, we also report the combined topic difficulty metrics (e.g., FusedD) using the (11) and the above two difficulty metrics. The topics in Table 5 are presented in descending order based on the FusedD value.

Table 5 Difficulty for getting an accepted answer for LCSD topics
Full size table

Topic Message Queue under Platform Adoption category is the most difficult topic to get an accepted answer in terms of FusedD value. Most median hours to get accepted answers (21). This topic contains discussion about general micro-service architecture (i.e., producer and consumer) and well as LCSD platform-specific support for these architectures. This is why this topic is also second most popular topic. Library Dependency Mngmt topic from Platform Maintenance is the second most difficult topic to get an accepted answer. Around 70% of its questions do not have any accepted answers. This topic concerns different troubleshooting issues about library and decencies of the system, server configuration, different library version compatibility issues. Web-Service Communication topic from Platform Adoption is the third most difficult topic. It has a long median wait time (around 20 hours) to get an accepted answer. This topic contains discussions about service-to-service communication via web service description language, HTTP REST message, and Windows Communication Foundation.

The topics that contain discussion about general software development (not specific to LCSD platforms) are the least difficult topics to get an accepted answer. For example, topic SQL CRUD under Data Storage category is the least difficult topic in terms of FusedD value (e.g., 0.5). This contains database CRUD related queries, and advanced queries too, such as inner join, nested join, aggregate. This also contains discussion about Object query language, which is a high-level wrapper over SQL. Topic SQL CRUD and SQL Syntax Error from the Data Storage category are two of the least difficult topics in terms of median hours to get accepted answers. Topic Pattern Matching and SQL CRUD are two of the least difficult topics in terms of questions without accepted answers.

Alternatively, topics that are specific to LCSD platforms are the most difficult topics. Four out of five most difficult topic belongs to Platform Adoption Categories. These questions can be popular as well as difficult. For example, LCSD-related Third-Party Integration related topic eSignature is the least popular topic from Table 4, is the most difficult topic in terms of questions without accepted answers (71%). Topic Platform Related Query is in the mid-range in terms of difficulty but most popular to get an accepted answer.

Correlation Between Topic Difficulty and Popularity Here we want to explore if there is any positive or negative relationship between topic popularity and difficulty. For example, Message Queue is the most difficult and, at the same time second most popular topic to get an accepted answer in terms of FusedD and FusedP metrics. Platform Related Query is the most popular but mid-range difficult topic.

Table 6 shows six correlation measures between topic difficulty and popularity in Tables 4 and 5. Three out of six correlation coefficients are negative, and the other three are positive and they are not statistically significant with a 95% confidence level. Therefore, we can not say the most popular topic is the least difficult to get an accepted answer and vice versa. Nonetheless, LCSD platform provides could use this insight to take necessary steps. Most popular topics should have an easy to access-able answer (i.e., least difficult).

figure l
Table 6 Correlation between the topic popularity and difficulty
Full size table

5 Discussions

During our analysis, we observed that several LCSD platforms are more popular across the topics than other platforms. We analyze our findings of LCSD topics across the top 10 most prevalent LCSD platforms in the dataset (Section 5.4). Finally, we discuss the implications of our study findings in Section 5.6.

5.1 Issues with Not Accepted Answers or Posts with Negative Score

In this paper, for topic modeling we used questions and accepted answers only. We did not consider the posts with negative score too because of the following observations. (1) Many other similar empirical studies on Topic modeling on SO posts also considered the questions and accepted answers only, e.g., IoT developers discussions in SO (Uddin et al. 2021b), big data related discussion (Bagherzadeh and Khatchadourian 2019), concurrency related topics (Ahmed and Bagherzadeh 2018), mobile app development (Rosen and Shihab 2015). (2) A significant number of studies (Asaduzzaman et al. 2013; Ren et al. 2019; Ponzanelli et al. 2014; Yang et al. 2016a) report quality of questions and unaccepted answers in SO are questionable and therefore it is quite a standard practice for SE researchers to consider the accepted answers in SO only. For example, In https://stackoverflow.com/questions/7504057/ (Fig. 12) a user asks question about python code/package to connect and retrieve data from Salesforce. The accepted answer https://stackoverflow.com/answers/7504244/ provides a relevant python code snippet the unaccepted answer https://stackoverflow.com/answers/34055640/ provide resource link for a command line tool which may be relevant but exactly not what the user asked for. (3) Negative scored questions are typically incorrectly tagged (e.g., https://stackoverflow.com/questions/4862071/, https://stackoverflow.com/questions/21377026/, https://stackoverflow.com/questions/37371712/), duplicates (e.g., https://stackoverflow.com/questions/12282151/, https://stackoverflow.com/questions/48121405/), lack a detailed problem description (e.g., https://stackoverflow.com/questions/25691340/, https://stackoverflow.com/questions/1974480/, https://stackoverflow.com/questions/50666660/), lack correct formatting (e.g., https://stackoverflow.com/questions/32208310/). For instance, in https://stackoverflow.com/questions/51635004/ (Fig. 12) a user inquires about an error encountered when attempting to contact a Zoho API. However, crucial important information such as an issue code or error message is lacking from the question description. In https://stackoverflow.com/questions/4862071/, an inexperienced user inadvertently tagged a question about the Oracle Apex platform with the Salesforce tag. We, therefore, choose not to include questions with a negative score or unaccepted answers. We also provide potentially missing out some insights for this choice in the threats to validity section (Section 6).

Fig. 12
figure 12

SO questions with a negative score or unaccepted SO answer

Full size image

5.2 Discontinued Low-Code Platforms and Future Trends

From our analysis on Section 4.2 we see the evolution of LCSD platforms, especially from 2012. According to our data, we can see the discontinuation of some low-code platforms but they are usually soon replaced by new low-code/no-code services. For example, In Jan 2020, Google announced the discontinuation of Google App Maker (2020) by 2021 (2021). But, shortly thereafter, Google announced a “no-code” platform called “AppSheet” (2021) and promoted their fully managed serverless platform called AppEngine (2021) to create web application promoting low-code approach. Microsoft and Amazon are also competing for superior low-code/no-code platforms with the emergence of new low-code service platforms such as Microsoft Power FX (2021), Amazon Honeycode (2022), AWS Amplify Studio (2022). The low-code approach is attracting increasing interest from traditional businesses, particularly during the pandemic (2022).

5.3 LDA Parameter Analysis

In this study, we applied LDA topic modelling, which employs Dirichlet distribution, to identify practitioners’ discussions on low-code. As described in details in Section 3.2, we followed the industry standard to configure the parameters and hyperparameters and also followed the industry recommendation to manually annotate the topics as described in Section 4.1 in order to avoid sub-optimal solutions (De Lucia et al. 2014). Following similar studies (Uddin et al. 2021b; Abdellatif et al. 2020; Han et al. 2020) we use the use the coherence score of of each model for different values of K. However, since LDA itself is probabilistic in nature (Agrawal et al. 2018) and can produce different results different runs on the same low-code dataset. In order to mintage this problem, we run our LDA model three times and compare the optimal number of topics. Figure 13 shows the result of different coherence score for different values of K. Moreover, we can see after reaching highest coherence values for K = 45 the overall coherence score decreases as the value of K increases.

Fig. 13
figure 13

The different coherence values for varied K on different runs

Full size image

5.4 The Prevalence & Evolution of Top Ten LCSD Platforms

Our analysis of the evolution of topic categories (see Section 4.2) shows that there is an overall increase in the number of new questions across the topics in SO. Our SO dataset is created by taking into account the LCSD platforms. In Fig. 14, we show how the 10 LCSD platforms evolve in our SO dataset over the past decade based on the number of new questions. Salesforce (2022) is the biggest and one of the oldest LCSD platforms (released in 1999) in our dataset with around 30% of all questions followed by Lotus Software (2022), Oracle Apex (2022), Microsoft powerapps(Microsoft Power Apps Platform Overview 2022). Among these platforms, IBM Lotus Software was quite popular during the 2014s and gradually lost its popularity, and IBM finally sold it in 2018. Salesforce platform has been the most popular platform in terms of SO discussions since 2012. Our graph shows that these other three platforms, especially Microsoft Powerapps, are gaining lots of attention during the pandemic, i.e., early 2020.

Fig. 14
figure 14

The evolution of top ten LCSD platforms discussions over time

Full size image

We provide more context for these platforms in Fig. 15 by illustrating the distribution of our observed five topic categories across the top ten LCSD platforms. We can see that Powerapps have the most number of queries in the Customization and Platform Adoption category. This happens because Powerapps is a relatively new LCSD platform (released in 2016) and it is gaining more and more attention from the community, and thus there are more queries such as business logic implementation (i.e., https://stackoverflow.com/questions/61685582/), connect Powerapps to a database in https://stackoverflow.com/questions/61611950/, user permission (i.e., https://stackoverflow.com/questions/61838119/). We can also see that older platforms such as Salesforce and Oracle Apex have more queries regarding Platform Maintenance, Third-Party Integration topic category. Practitioners ask many different questions regarding these platforms such as deployment-related (e.g., “Deploying a salesforce.com flex app without visualforce” in https://stackoverflow.com/questions/6614226/), third-party API integration (e.g., “Google Map Integrated with Salesforce is Blank” in https://stackoverflow.com/questions/9028682/), maintenance deployment (e.g., “Salesforce deployment error because of test class failure” in https://stackoverflow.com/questions/9171945/), interactive report in https://stackoverflow.com/questions/9700660/, customization with JSON https://stackoverflow.com/questions/9833992/, “what is dashboard?” in https://stackoverflow.com/questions/10911269/ and Oracle Apex how to use platform in https://stackoverflow.com/questions/9438695/. Platform Adoption is a prevalent topic category in the Powerapps, ServiceNow, and Tibco platforms. We also notice that the Data Storage category is quite popular in Filemaker and Lotus Software. Interestingly we see that Zoho Creator LCSD platform around 60% questions belong to Third-party API integration (especially email configuration https://stackoverflow.com/questions/48865565/). This data sheds light on the popular discussion topics of more recent and earlier LCSD platforms.

Fig. 15
figure 15

The distribution of topic categories across top ten LCSD platforms

Full size image

5.5 The Case of “Aggregating” Data Across Multiple LCSD Platforms

In this study, we analysed 38 LCSD platforms and these platforms have distinct characteristics and challenges. Our goal is to offer researchers and practitioners a comprehensive overview of the LCSD domain as a whole, as opposed to focusing on a single LCSD platform. Hence, we integrated the data from all of these platforms. For instance, Fig. 14 demonstrates that some of the most popular platforms such as Salesforce, Oracle Apex, and Microsoft Powerapps have more questions in SO than other LCSD platforms. Figure 15 demonstrates that questions across these platforms over different topic categories differs slightly. However, Fig. 16 shows that questions for Application Customization and Platform Maintenance topic category for top ten platforms vs others remain about the same at around 30% and 13% respectively. Popular platforms have more questions related to Third Party API integration (15%) than others (4%). The top ten platforms have relatively fewer questions (15%) in Data Storage (23% vs 29%) and Platform Adoption (19% vs 24%) Category compared to other platforms. Overall, we found that the observed topics are found across all the platforms that we studied. Given the popularity of some platforms over others, it is understandable that those platforms are discussed are more and as such some platforms can have more coverage (in terms of number of questions) in a topic over other platforms. However, the prevalence of all platforms across each topic shows that the topics are generally well-represented across the platforms.

Fig. 16
figure 16

The percentage of questions distributed across five topic categories for the top ten LCSD platforms versus the rest of the platforms

Full size image

5.6 Implications

In Table 7, we summarize the core findings of our study and provide recommendations for each findings. The findings from our study can guide the following three stakeholders: (1) LCSD platform Providers to improve the documentation, deployment, and maintenance support, (2) LCSD Practitioners/Developers to gain a better understanding of the trade-offs between rapid development and customization constraints, (3) Community of LCSD Researchers & Educators to have a deeper understanding of the significant challenges facing the broader research area to make software development more accessible. We discuss the implications below.

Table 7 The summary of our core findings and recommendations
Full size table

In this empirical study, we infer implications and recommendations based on our observation of practitioners’ discussion in SO. So further validation from the developers’ survey can provide more insight. However, the diversity of the low-code platforms and topics makes it non-trivial to design a proper survey with representative sample of LCSD practitioners. Therefore, the findings can be used to design multiple LCSD related surveys focusing on different low-code topics and platforms (Fig. 17).

Fig. 17
figure 17

The popularity vs. difficulty bubble chart for 40 LCSD-related topics

Full size image
LCSD Platform Vendors. :

In order to better understand the issues of LCSD, we present a bubble chart with difficulty and popularity of different aspects of LCSD such as Topic Category in Fig. 11, Types of questions in Fig. 19 and agile SDLC phases in Fig. 18. These findings coupled with the evolution of LCSD platforms (14) and discussions (7) shows that Customization and Data Storage related queries are more prevalent, with the majority of these queries occurring during Implementation agile SDLC stage. However, one of our interesting findings is Platform Adoption related queries are increasing in popularity. LCSD practitioners find LCSD platform infrastructure and server configuration-related quires tough and popular during the Deployment and Maintenance phase. The top five most challenging topics belong to Platform Adoption and Maintenance topic category.

Fig. 18
figure 18

The popularity vs. difficulty bubble chart for low-code software development life cycle phases

Full size image

Many new practitioners make queries regarding LCSD platforms, learning resources, basic application and UI customization, and how to get started with this new emerging technology. Figure 17 shows that Platform Related Query topic is the most popular among LCSD practitioners. We find that Documentation related queries are both top-rated and challenging. Our findings also suggest that many practitioners still face challenges during testing, especially with third-party testing tools like JUnit (in https://stackoverflow.com/questions/9811992/) and troubleshooting. Consequently, many of the questions on this topic remain unanswered. It reveals that to ensure smooth adoption of the LCSD platforms, providers should provide better and more effective documentation and learning resources to reduce entry-level barriers and smooth out the learning curve.

LCSD Practitioners/Developers. :

Gartner (2022) estimates that by 2022, more than half of the organizations will adapt LCSD to some extent. Additionally, our analysis reveals a rising trend for LCSD approaches, particularly during Covid-19 pandemic (Fig. 6). We can also see that new LCSD platforms such as Microsoft Powerapps are gaining many developers’ attention. LCSD platform enables practitioners with diverse experience to contribute to the development process even without a software development background. However, our finding shows that practitioners find debugging, application accessibility, and documentation challenges. Hence, the practitioners should take the necessary steps to understand the tradeoffs of LCSD platforms’ features deeply. The project manager should adopt specific strategies to customize, debug, and test the application. For example, many practitioners struggle with general Third-Party API integration and database design and query. We find that DevOps-related tasks such as CI/CD, Server configuration, and monitoring-related queries are most challenging to the practitioners. So, a well-functioning LCSD team should allocate time and resources to them. It provides valuable insights for project managers to manage resources better (i.e., human resources and development time).

Figure 18 shows that Maintenance is the most popular development phase, followed by Deployment, and Testing is the least popular SDLC phase. Similarly, the figure also shows that questions asked in deployment phase are the most difficult followed by Maintenance. Implementation, Requirement analysis and planning, Application design phase are in the middle range in terms of popularity and difficulty spectrum. Thus, our analysis indicates that LCSD practitioners face more broad and complex application maintenance and deployment-related challenges, on which LCSD platform vendors should concentrate their efforts. This finding can influence the decision-making process of LCSD developers and practitioners like prioritizing their efforts during the design, development, and deployment of software that uses LCSD platforms. For example, if sufficient support or tools are not available for scalable usage and deployment of an LCSD platform, developers may look for alternatives that have better deployment and maintenance support.

One fundamental shortcoming of LCSD platforms is that their abstraction and feature limitations can make customization and debugging extremely difficult. Additionally, managed cloud platforms make data management and deployability more challenging (Sahay et al. 2020; Luo et al. 2021). The findings in this study help to present some strengths and limitations of the overall LCSD paradigm, which complements the findings of other studies (Sahay et al. 2020; Alsaadi et al. 2021; Luo et al. 2021; Waszkowski 2019; Adrian et al. 2020; Ness and Hansen 2019). The analysis could assist LCSD teams in selecting the appropriate LCSD platforms, which is critical for future success.

LCSD Researchers & Educators. :

The findings of this study have many implications for researchers and educators of LCSD platforms and the border research community to improve the software development process. We discover that What-type and How-type questions are popular among LCSD practitioners. They also find them challenging because of adequate usable documentation. Thus, practitioners ask questions about certain limits or how to implement certain features, and in the accepted answer, some other user simply points to the official documentation page (e.g., “Domino Data Service API Documentation” in https://stackoverflow.com/questions/59739877/ and https://stackoverflow.com/questions/5806293/). Many of the challenges faced by low-code petitioners are similar to traditional software developers. So, researchers from border software engineering domain can contribute to improving aspects such as improving documentation (Bayer and Muthig 2006; Khan et al. 2021a; Bhat et al. 2006), improving API description usage (Uddin and Robillard 2015a; Uddin et al. 2021a) and make it more accessible to general practitioners. In the Customization and Data Storage topic category, we find practitioners asking help in generic programming queries, database design, and file management. So, research on those topics will also help the adoption of LCSD. Some LCSD platforms provide great in-build support for unit and functional testing. However, we find around 2.1% of questions belong to Testing topic. Most of these LCSD platforms heavily rely on cloud computing, and thus research improvement of server configuration and library management, i.e., DevOps (Zhu et al. 2016) in general, will aid in better platforms. On the other hand, educators can focus their efforts on making the learning resources on Automatic testing, Server config. and DevOps practices such as CI/CD more accessible to the citizen developers.

Figure 19 shows What-type of posts are most popular, followed by Why-type, How-type, and Others-type. Additionally, it demonstrates that the most challenging question type is Why-type, followed by What-type, How-type, and others. So, although How-type questions are most dominant, the details types (i.e., Why-type, What-type) of questions are more popular and challenging. This analysis implies that LCSD practitioners have a harder time finding detailed information regarding different platform features. As a result, LCSD platform providers should improve their documentation. Intuitively, How-type questions can be answered with better documentation for LCSD platforms. Given the official API documentation can often be incomplete and obsolete (Uddin and Robillard 2015b; Khan et al. 2021b) and given that our research shows that LCSD developers use SO to ask questions about various topics, LCSD researchers can develop techniques and tools to automatically improve the documentation of LCSD platforms by analyzing SO questions and answers. Indeed, existing research efforts show that solutions posted in SO can be used to improve API documentation by supporting diverse development tasks and programming languages (Chakraborty et al. 2021; Uddin et al. 2019, 2020a, 2020b; Uddin and Khomh 2017b, 2017c, 2019).

Fig. 19
figure 19

The popularity vs. difficulty of different types of LCSD-related questions

Full size image

We find that the LCSD paradigm’s challenges can be different from traditional software development (Sahay et al. 2020). Simultaneously, researchers can study how to provide better tools for practitioners to customize the application. Security is an open research opportunity for such platforms as a security vulnerability in such platforms or frameworks could compromise millions of applications and users (Lin et al. 2020). Researchers can develop better testing approaches to ensure faster development and dependability. Educators can also benefit from the results presented in Table 4 and Fig. 17 to prioritize their focus on different topics such as Library Dependency Mngmt, Web-Service Communication, Asynchronous Batch Jobs, Testing, Dynamic Form Controller.

6 Threats to Validity

Internal validity threats, in our study, relate to the authors’ bias while conducting the analysis as we have manually labeled the topics. We mitigate the bias in our manual labeling of topics, types of questions, and LCSD phases by consulting the labels among multiple authors and resolving any conflicts via discussion. Four of the authors actively participated in the labelling process. The first author reviewed the final labels and refined the labels by consulting with the second author.

Construct Validity threats relate to the errors that may occur in data collection, like identifying relevant LCSD tags. To mitigate this, we created our initial list of tags, as stated in Section 3, by analyzing the posts in SO related to the leading LCSD platforms. Then we expanded our tag list using state-of-art approach (Bagherzadeh and Khatchadourian 2019; Abdellatif et al. 2020; Ahmed and Bagherzadeh 2018; Rosen and Shihab 2016). Another potential threat is the topic modeling technique, where we choose K = 45 as the optimal number of topics for our dataset B. This optimal number of topics has a direct impact on the output of LDA. We experimented with different values of K following related works (Abdellatif et al. 2020; Bagherzadeh and Khatchadourian 2019). We used the coherence score and manual examination to find K’s optimal value that gives us the most relevant and generalized low-code related topics (Alamin et al. 2021; Uddin et al. 2021b; Abdellatif et al. 2020).

External Validity threats relate to the generalizability of our findings. Our study is based on data from developers’ discussions on SO. However, there are other forums LCSD developers may use to discuss. We only considered questions and accepted answers in our topic modeling. We also had the option of choosing the best answer. In SO, the accepted answer and best answer may be different. Accepted answer is the one approved by the questioner while the best answer is voted by all the viewers. as discussed in Section 5.1 it is quite difficult to detect if an answer is relevant to the question or not. Thus We chose the accepted answer in this study because we believe that the questioner is the best judge of whether the answer solves the problem or not. Even without the unaccepted answers, our dataset contains around 38K posts (27K questions + 11K accepted answers). This also conforms with previous works (Uddin et al. 2021b; Abdellatif et al. 2020; Alamin et al. 2021; Yang et al. 2016a). Some novice practitioners post duplicate questions, assign incorrect tags, and provide inadequate descriptions, which receives an overall negative score from the community. To ensure that topics contain relevant discussion of high quality, we only use posts with non-negative scores. Nevertheless, we believe using SO’s data provides us with generalizability because SO is a widely used Q&A platform for developers. However, we also believe this study can be complemented by including the best answers to the questions in SO, as we discussed earlier, including discussions from other forums, surveying, and interviewing low-code developers.

7 Related Work

We previously published a paper at the MSR 2021 based on an empirical study of LCSD topics in SO (see (Alamin et al. 2021)). We compared the findings of this paper against our previous paper in Section 1. Other related work can broadly be divided into two categories: SE (Software Engineering) research on/using (1) low code software development (Section 7.1), and (2) topic modeling (Section 7.2).

7.1 Research on Low Code Software Development and Methodologies

LCSD is a new technology, with only a handful of research papers published in this field. Some research has been conducted on the potential applications of this developing technology in various software applications (Fryling 2019) or for automating business process in manufacturing (Waszkowski 2019), healthcare (Ness and Hansen 2019; Woo 2020), Digital transformation (Phalake and Joshi 2021), Industrial engineering education(Adrian et al. 2020), IoT systems using LCSD (Ihirwe et al. 2020a). Di Sipio et al. (2020) present the benefits and future potential of LCSD by sharing their experience of building a custom recommendation system in the LCSD platform. Kourouklidis et al. (2020) discuss the low-code solution to monitor the machine learning model’s performance. Sahay et al. (2020) survey LCDP and compare different LCDPs based on their helpful features and functionalities. Khorram et al. (2020) analyse commercial LCSD platforms and present a list of features and testing challenges. Zhuang et al. (2021) created a low-code platform called EasyFL where researchers and educators can easily build systems for privacy-preserving distributed learning method. Ihirwe et al. (2020b) analyse 16 LCSD platforms and identifies what IoT application-related features and services each platform provides. All of these studies compare a single LCSD platform and its support and limitations for various sorts of applications (Alonso et al. 2020), rather than taking a holistic picture of the difficulties that the broader community faces.

There are also some studies where researchers proposed different techniques to improve LCSD platform such as Overeem and Jansen (2021) on LCSD platform’s impact analysis, Jacinto et al. (2020) improve testing for LCSD platforms.

Additionally, there are some studies that describe the difficulties faced by LCSD practitioners. The main research methodology and objectives of these studies, however, are significantly different from this study. Luo et al. (2021) analyse the LCSD platform’s characteristics including programming languages used, major implementation units, supporting technologies, applications being developed, domains, etc., along with the benefits, limitations, and challenges by collecting relevant posts from SO and Reddit. In this study, we use tag-based approach to find relevant LCSD-related posts which is much more reliable than text-based searching. Furthermore, the SO related discussion used in this study is significantly larger and our research objective about LCSD platforms challenges are quite different. Lethbridge (2021) discuss experiences with several low-code platforms and provide recommendations focusing on low-code platforms enabling scaling, understandability, documentability, testability, vendor-independence, and the overall user experience for developers as end-users who do some development. Dahlberg (2020) and Alsaadi et al. (2021) Surveyed on factors hindering the widespread adaptation of LCSD by interviewing LCSD developers or conducting a survey. To the best of our knowledge, ours is the first empirical study of LCSD platforms based on developers’ discussions from Stack Overflow, and hence our findings complement those of other studies.

7.2 Topic Modeling in Software Engineering

Our motivation to use topic modeling to understand LCSD discussions stems from existing research in software engineering that shows that topics generated from textual contents can be a good approximation of the underlying themes (Chen et al. 2016; Sun et al. 2015b, 2016). Topic models are used recently to understand software logging (Li et al. 2018) and previously for diverse other tasks, such as concept and feature location (Cleary et al. 2009; Poshyvanyk et al. 2007), traceability linking (e.g., bug) (Rao and Kak 2011; Asuncion et al. 2010), to understand software and source code history evolution (Hu et al. 2015; Thomas et al. 2011, 2014), to facilitate code search by categorizing software (Tian et al. 2009), to refactor software code base (Bavota et al. 2014), as well as to explain software defect (Chen et al. 2012), and various software maintenance tasks (Sun et al. 2015a, b). The SO posts are subject to several studies on various aspects of software development using topic modeling, such as what developers are discussing in general (Barua et al. 2014) or about a particular aspect, e.g., concurrency (Ahmed and Bagherzadeh 2018), big data (Bagherzadeh and Khatchadourian 2019), chatbot (Abdellatif et al. 2020).

In particular, SO posts have been used in various studies where the researchers analysed topics for that particular domains. For instance, SO posts has been used to study developers challenges in IoT (Uddin et al. 2021b), big data (Bagherzadeh and Khatchadourian 2019), chatbots (Abdellatif et al. 2020) and so on. The distributions and the nature of these posts differs. As SO is arguably the most popular public forum for developers, the analysis of these domains’ characteristics may help us identify the SO community better. Therefore, a systematic analysis of these domains is interesting. Following related studies(Uddin et al. 2021b), we use six metrics in this study: (1) Total #posts, (2) Avg views, (3) Avg favorite, (4) Avg score, (5) Percentage of questions without accepted answers, (6) Median hours to get accepted answers per domain. The first four metrics are popularity metrics and the last two are difficulty metrics.

In this study, we do not replicate the findings of the original study in our dataset. Rather we only report the findings from the original study. So following related work (Uddin et al. 2021b), we compared our LCSD-related discussions with other five domains: IoT (Uddin et al. 2021b), big data (Bagherzadeh and Khatchadourian 2019), security (Yang et al. 2016b), mobile apps (Rosen and Shihab 2015), chatbots (Abdellatif et al. 2020) and concurrency (Ahmed and Bagherzadeh 2018).

Table 8 provides an overview of the seven metrics. We can see that it has a greater number of SO posts than chatbot domains but fewer than the other five domains. There are two other studies on Blockchain (Wan et al. 2019) and deep learning (Han et al. 2020) where the total number of posts are 32,375 and 25,887, respectively. However, these two studies did not report the other metrics, so they are excluded from the Table. Although the LCSD-related discussion may have fewer posts than these other domains, as discussed in RQ3, this number is increasing rapidly.

Table 8 Comparing the popularity and difficulty metrics of different domains
Full size table

We can also observe that the LCSD domain shows similarities with IoT, Concurrency domain in terms of Avg. View count. Security and Mobile domain seem most popular in terms of Avg. Favourite count and LCSD rank lower in this metric. LCSD domains most resemble with IoT domain in terms of Avg. View, Avg. Favourite and Avg. Score. In terms of difficulty metrics percentage of posts without accepted answers, LCSD domain ranks lowest, which is good. However, it takes much longer to get accepted answers(0.7 hours for Mobile, 2.9 for IoT). Only the chatbot domain requires more time to get accepted answers (i.e., 5.3 hours in LCSD vs 14.8 hours in chatbot).

We further discuss the LCSD, IoT, and Chatbot domains with the distribution of different types of questions in Table 9. We find that LCSD domain is in the middle of IoT and Chatbot domains in terms of How-type (57%) compared to IoT (47%) and chatbot (62%). This signifies that LCSD domain practitioners ask more about implementation-related questions. In the Why-type question percentage of LCSD-related domains is lowest (14%) compared with IoT (20%) and chatbot (25%). This suggests that practitioners in the LCSD domain enquire about relatively modest troubleshooting issues. For What-type, we notice that the IoT domain dominates with 38% of questions, compared to the LCSD domain’s 12%. Practitioners of LCSD are less inquisitive about domain architecture and technologies compared to IoT domain. As a result of these analyses, we can see that LCSD domain practitioners exhibit several traits that distinguish them from practitioners in other domains, which LCSD vendors and educators should take into account.

Table 9 Comparative analysis of question types across different domains
Full size table

8 Conclusions

Low Code Software Development (LCSD) is a novel paradigm for developing software applications utilizing visual programming with minimum hand-coding. We present an empirical study that provides insights into the types of discussions low-code developers discuss in Stack Overflow (SO). We find 40 low-code topics in our dataset of 33.7K SO posts (question + accepted answers). We collected these posts based on 64 SO tags belonging to the popular 38 LCSD platforms. We categorize them into five high-level groups, namely Application Customization (30% Questions, 11 Topics), Data Storage (25% Questions, 9 Topics), Platform Adoption (20% Questions, 9 Topics), Platform Maintenance (14% Questions, 6 Topics), and Third-Party Integration (12% Questions, 5 Topics). We find that the Platform Adoption topic category has gained popularity recently. Platform Related Query and Message Queue topics from this category are the most popular. On the other hand, We also find that practitioners find Platform Adoption and Maintenance related topics most challenging. How-type questions are the most common, but our research reveals that practitioners find what-type and why-type questions more difficult. Despite extensive support for testing, deployment, and maintenance, our analysis shows that server configuration, data migration, and system module upgrading-related queries are widespread and complex to LCSD practitioners. Despite significant testing, deployment, and maintenance support, our analysis finds numerous and complex queries regarding server configuration, data migration, and system module updating. Our analysis finds that better tutorial-based documentation can help solve many of these problems. We also find that during the Covid-19 pandemic, LCSD platforms were very popular with developers, especially when it came to dynamic form-based applications. We hope that all of these findings will help various LCSD stakeholders (e.g., LCSD platform vendors, practitioners, SE researchers) to take necessary actions to address the various LCSD challenges. Since the growth indicates that this technology is likely to be widely adopted by various companies for their internal and customer-facing applications, platform providers should address the prevailing developers’ challenges. Our future work will focus on (1) getting developers’ feedback on our study findings based on surveys and developer interviews, and (2) developing tools and techniques to automatically address the challenges in the LCSD platforms that we observed.