Introduction

Recent years have seen a plethora of discussions and studies on how big data infrastructure, accrued through sensors, and associated information and communication technologies (ICTs) can help achieve sustainable development goals and improve the management of cities (Geertman and Stillwell 2020; Pan et al. 2019; Thakuriah et al. 2017; Khan et al. 2015; Caragliu et al. 2011). In planning, planning support systems (PSS) have been imagined and conceptualized to make use of big data and interactive interfaces to achieve smart goals (Babar and Arif 2017; Deal et al. 2017a; Allwinkle and Cruickshank 2011). Defined as geo-information technology-based instruments, PSS are dedicated to supporting those involved in planning in the performance of their specific planning tasks (Geertman 2006, 2017). More recently, studies have shown that with the advent of urban sensing and ubiquitous computing, and the gradual standardization of embedded location information within administrative datasets on urban activities, the promise of a smart city has led to an exponential increase in data by several orders of magnitude (Geertman and Stillwell 2020). Consequently, such enormous volumes of data, or big data, act as valuable input for PSS. By capturing, analyzing, and integrating this real-time and up-to-date data into various types of PSS, spatial analyses related to the realm of urban planning in the city (e.g., energy consumption, land use, traffic congestion, energy usage, and air quality) are improved substantially (Rathore et al. 2016; Khan et al. 2015). Besides, ICTs and big data can also be used to enhance PSS’s capabilities for extending and transforming somewhat conventional citizens’ participation practices, and for facilitating technology-facilitated communication and interplay between the formal political sphere (government) and the civil society sphere (citizens) (Jiang et al. 2019, 2020a; Khan et al. 2014). For instance, Pan et al. (2020) show that urban informatics augmented by new smart data enhance the sharing of information to the general public and make planning processes more participatory and democratic, especially for disadvantaged groups. Geertman and Stillwell (2020) indicate that web-based PSS enable more non-state actors (i.e., individuals and organizations that are not affiliated to the government, such as private sector entities, academic institutions, and non-governmental organizations) to build solutions that enable the delivery of content, services, and even applications over the HTTP protocol. Briefly, PSS enabled by smart ICTs help enhance reach and range by enabling information to be shared across different stakeholders and contribute to aspects of the planning process, including data collection and storage, data analysis and presentation, plan- and policy-making, interpersonal dialogue and debate, and policy implementation and administration (Jiang et al. 2020b; Pettit et al. 2018; Zhang et al. 2019; Thakuriah et al. 2017).

It should be noted, however, that in practice PSS have not fully achieved their potential, since their usefulness depends not only on the characteristics and capabilities of the tools themselves (Pan and Deal 2020; Geertman 2017; Russo et al. 2018; Pelzer 2015). It has been widely accepted that PSS need to be enhanced to better align the instruments with user requirements and the planning tasks and problems at hand, since the specific situations or contexts in which PSS are embedded have a significant influence on how PSS work in actual planning practice (Champlin et al. 2019; Geertman 2017; Geertman 2006). Here, context indicates the real circumstance or situation in which a PSS tool is embedded in planning practices. Empirical studies have demonstrated the influence of contextual factors on PSS usefulness. For instance, Pelzer (2017) found that existing organizational hierarchies, the timing of the policy process, and the users’ characteristics (disciplinary background, existing habits, etc.) affect how the role of a PSS is perceived, implemented, and evaluated. Zhang et al. (2019) point out that digital literacy has prevented many ordinary people from engaging in web-based participatory planning. As Jiang et al. (2020c), Geertman (2017, 2006), and te Brömmelstroet (2010) argue, what can be achieved with a PSS, and the meaning of those achievements, largely depend on how the planning context is treated.

However, despite the importance of contextual factors in influencing PSS usefulness in practice, research on PSS has concentrated more on aspects of the tools than on understanding the interactions between the tools and the contexts in which they are used (McEvoy et al. 2019; Champlin et al. 2019). This shortcoming is well recognized in the field of PSS, which calls for tools to be studied in the “real world, context-rich environment” (te Brömmelstroet 2013:306). Rather than taking a normative view, it embraces a more pragmatic attitude toward the role of ICT: To what extent can the implementation of ICT in planning become more effective and useful? (Deal et al. 2017b; Geertman 2017). Consequently, research into the supportive role of PSS in the realm of smart cities has recently been advanced to embrace such a question: What kinds of ICT are, or should be, implemented by what kinds of stakeholders in which types of planning situations, contexts, or circumstances? (Russo et al. 2018; Pelzer et al. 2015; Biermann 2011; Vonk 2006). It has to be noted, however, that the significance of the proposed guidelines for improving PSS usefulness in practice is still limited by a lack of studies on theorizing the different contextual factors and their effects in practice.

Thus, the aim of this paper is to draw attention to the critical influence of contextual factors on the usefulness of PSS in smart cities. An ordinal regression model was applied to data gathered through an international questionnaire, to quantify the impact of the identified contextual factors on PSS usefulness. It should be noted that this study was an extension to the work by Geertman (2006), te Brömmelstroet (2013), and Pelzer (2017), who argue that further study could systematically analyze the effects of the contextual factors on PSS usefulness in practice. Based on this, Section 2 operationalizes the key terms describing earlier research and the various indicators related to the role of context in PSS usefulness. Section 3 introduces the methodology. Section 4 describes the main findings from the questionnaire. This is followed by a discussion in Section 5. The paper ends with a conclusion.

Context and PSS Usefulness

The study by Pelzer et al. (2014) found that the added value of PSS is often conceived as the focal point, since the questions permeating these studies not only reveal the value of PSS, but also contribute to supporting planning in a better way. From a general technological viewpoint, Nielsen (1994:24) conceptualizes the added value of a system as “usefulness,” which indicates “the issue of whether the system can be used to achieve some desired goal.” As te Brömmelstroet (2013) argues, the usefulness of PSS can be measured at both the process level and the outcome level. And Pelzer et al. (2014) highlight that an evaluation of the usefulness of a PSS tool should focus on at least three dimensions, namely individual, group, and outcome. Based on this, they identify seven major types of usefulness that a PSS tool can provide.

In practice, scholars and professionals have developed a wide variety of applications to support all kinds of planning actions, ranging from highly strategic to operational actions (te Brömmelstroet 2013). However, what is remarkable about PSS is its long-standing implementation gap, that is, a discrepancy between supply and demand: Despite the availability of a growing number and variety of potentially valuable PSS instruments, planning practitioners are rather hesitant to buy, implement, or apply them (Geertman 2006, 2017). To find the main bottlenecks to increasing the usefulness of PSS in planning practice, numerous authors have identified that the support role of PSS is closely related to the context in which a PSS is applied (Luque-Martín and Pfeffer 2020; McEvoy et al. 2019; Zhang et al. 2019; Vonk and Geertman 2008; Vonk et al. 2005; Geertman 2006). According to Pelzer (2017), context influences both the extent to which a PSS is useful and the kind of usefulness that is achieved. For instance, at an early stage of a policy process, learning about the object might be the prime usefulness (cf. te Brömmelstroet 2010; Goodspeed 2013), whereas at a later stage efficiency might be more important. In addition, different planning styles affect how the technological intelligence is organized and operates; for example, the ICT-enabled centralized planning process in which the entire set of intelligent devices is arranged hierarchically and steered into one center, versus ICT-facilitated participatory planning processes in which new technologies enable various stakeholders to participate in urban issues. As Luque-Martín and Pfeffer (2020) summarize, local context specifics are one of the main factors that define the PSS-based potential solutions. Geertman (2006) provides a comprehensive conceptual discussion on the different contextual factors. In his framework, six major contextual factors are argued to influence the potential roles of PSS in planning practice, namely characteristics of the technology, content of the planning issue, user characteristics, characteristics of the planning process, planning style, and political context (see Fig. 1). The present study applied and operationalized these six contextual factors to further examine the effects of contextual factors on PSS usefulness in practice.

Fig. 1
figure 1

Contextual factors influencing PSS usefulness (framework based on Geertman (2006: 867); the indicators describing each variable are explained in section 2.1)

Full size image

Contextual Factors

Characteristics of the Technology

Regarding the characteristics of the technology itself, Geertman (2006) argues that planning support tools are contextual in that the support role of PSS proves to be more evident in operational decision-making than at the level of strategic decision-making. From a technological innovation perspective, the technology itself as a contextual factor can be understood from two variables: functionality and usability. Functionality “is the question of whether [the information-handling capabilities] of the system in principle can do what is needed” (Nielsen 1994:24). Since the planning tasks vary in different planning practices, a PSS tool in one specific project is unlikely to act in the same way in another project. Thus, an effective way to achieve planning support is to apply the information-handling capabilities of a PSS to support specific professional spatial-planning tasks (Luque-Martín and Pfeffer 2020; McEvoy et al. 2019; Zhang et al. 2019). Several authors (e.g., Jiang et al. 2020a; Vonk 2006) categorized these information-handling capabilities into three groups: informing, communicating, and analyzing & designing.

  • Informing indicates the functional support for transferring information in one direction only.

  • Communicating describes the functional support for facilitating communication and discussion between those involved in the governance process through supporting flows of information between them.

  • Analyzing & designing indicates the functional support for examining spatial patterns of human behavior and facilitating the perception, production, and presentation of design ideas.

Usability refers to “how well users can use that [functionality]” (Nielsen 1994:24). It indicates the goodness of fit between the capabilities of the user and the functionalities offered by the technology (Jiang et al. 2020a). In general, a high level of usability will lead to the more usefulness of a PSS. In literature, although a range of usability indicators have been identified (Pelzer 2015), four key indicators are widely used to describe the usability of a PSS: transparency, user friendliness, interactivity, and flexibility (Pan and Deal 2020; Champlin et al. 2019; te Brömmelstroet 2017; te Brömmelstroet 2012; te Brömmelstroet 2010; Vonk et al. 2005).

  • Transparency describes the extent to which the underlying models and variables of the PSS are visible to users.

  • User friendliness is the extent to which participants are able to use the tool themselves.

  • Interactivity refers to the extent to which the tool can directly respond to the users’ questions and suggestions.

  • Flexibility indicates the extent to which the tool can be applied to different planning tasks.

Together, these functionality and usability indicators delineate the specific characteristics of the technology (PSS) itself.

Content of the Urban Issue

According to Rittel and Webber (1984), a range of strategic, nonroutine planning issues can be said to be ill-structured or at best semi-structured. New PSS are primarily developed to provide new capabilities to gain insights into genuinely unique spatial issues. However, it should be noted that many of the urban issues that PSS tools attempt to solve are typically ill-structured or ill-defined by nature (Klosterman 1997; Cats-Baril and Huber 1987). The consequence is that there is no clear procedural or predetermined way to solve a specific urban issue; instead, it requires diverse and innovative solutions. Because of this, the PSS—which usual promotes a false idea of value-free and objective knowledge of urban issues—exacerbates technocratic reductionism (Söderström et al. 2014) and masks those urban conflicts and issues that are not visible in the digital representations (e.g., congestion, unemployment, pollution, and diseases peculiar to the urban way of life) (Verrest and Pfeffer 2019). Hence, to improve the appropriateness of dedicated PSS tools, it is vital to distinguish the different attributes of the urban issue. In this study, we applied the variable “types of urban issues” to operationalize the content of the urban issue. According to Campbell (1996), economic issues, social issues, and environmental issues are the three main types of urban issues.

User Characteristics

User characteristics are contextual factors in that they are a useful source of information that helps designers to foster an understanding of target users and the product under development (Chesbrough 2006; Geertman 2006). For each category of user, the demands and requirements regarding the content of the functionalities needed differ considerably. For instance, in the profession of planning practitioner, there is a significant difference between more design-oriented practitioners and research-oriented practitioners: The former need more functional support that can stimulate their imagination and lead to original and innovative ideas, whereas research-oriented practitioners are more concerned about how a PSS functionality can gather, store, analyze, model, and visualize data in a digital representation (Brail and Klosterman 2001). As Geertman (2006: 868) argues, “professional protectionism plays a role of importance.” In addition, attributes of users such as competency, skill, knowledge, qualification, and proficiency level also affect the performance of PSS in practice (Zhang et al. 2019; Pelzer 2017; Holsapple et al. 2005). Without a doubt, an inexperienced newcomer (newbie) differs from a person who has comprehensive and authoritative knowledge or skill in terms of using PSS (expert) (Vonk et al. 2005). Finally, Vonk and Geertman (2008) argue that user attitude is also responsible for promoting the supportive role of PSS in their field. Users with a rational mindset and a positive attitude toward PSS will usually be early adopters and be more enthusiastic about PSS innovations and their added value. Therefore, a user-oriented approach in developing and implementing PSS requires a deeper understanding of user characteristics related to the expected and unexpected problems that users face when interacting with particular PSS tools. In this study, the three previously mentioned variables—profession, expertise level, and user attitude—were applied to operationalize the user characteristics.

Characteristics of the Planning Process

The characteristics of a planning process are the qualities or features that are regarded as inherent parts of a plan. For instance, plans are made for a period of, say, five, ten, or more years. Therefore, in composing a plan and getting it politically approved, PSS users may face time pressure or strict deadlines (Geertman 2006; Vonk 2006). Then, active uptakers are responsible for signaling promising supportive roles of PSS in their field and bringing these to the attention of other potential users (Vonk and Geertman 2008). For instance, geo-information specialists have the important role of scanning the organization’s external environment, signaling PSS developments, and evaluating their usefulness to the planning organization (Vonk 2006). Besides, funding is also crucial for providing continuous support and maintenance of the PSS being used in practice (Ismail 2008; Hutchison et al. 2006). Thus, three variables are used to describe the characteristics of the planning process: time pressure, active uptaker, and funding.

Planning Style

A planning style is understood as the leading form of planning during a certain period of time (Pelzer 2015). It represents time-bound normative opinions as to the way in which the planning job should be performed (Healey 1997). However, it should be noted that the influence of planning style on PSS usefulness changes along with changes in time and place. For instance, the dominant rational comprehensive model in the 1960s led to the design and implementation of a range of large-scale computer-based systems (e.g., large-scale metropolitan land use, integrated municipal information systems, and urban transportation models) to solve some of the large and urgent urban issues (Chadwick 1977). Thus, the usefulness of PSS at that time was mainly related to gaining insights into urban issues. Then, in the 1990s, PSS were used to facilitate interpersonal communication and community-based debates because of the rise of participatory and collaborative planning (Pelzer 2015; Pettit 2005). In the present study, planning styles were categorized as either “closed planning” or “interactive planning” (Jiang et al. 2020b; Healey 1997).

  • In closed planning, the government and its agent planner act as the prime stakeholders in the planning process.

  • In interactive planning, also non-state actors—like citizens and the private sector—are important and involved in the planning process.

Political Context

As a decision-making process, each planning process should be aware of the broader political context in which it is embedded (Jiang et al. 2019; Lin 2018; Biermann 2011). This context has specific mechanisms that influence how technology is organized in planning practice. For instance, in democratic Western countries, there is a general consensus that the increasing number of crowdsourcing platforms, social media, smartphones, portals, and planning support systems should be applied to facilitate e-participation, e-democracy, and wider collaboration for collective problem-solving (Geertman and Stillwell 2020; Lin 2018). In contrast, in some societies that are tightly controlled by the government, the use of innovative technologies in planning is more oriented toward improving the efficiency and effectiveness of the planning process, for example, to gain a better understanding of the urban issue and inform the public of the plan (Long et al. 2011). According to some authors, the influence of the political context can be described as “political pressure” (Stanilov 2007), which reflects comprehensive persuasion by, the influence of, or even intimidation from members of the political planning systems.

PSS Usefulness

As explained by Pelzer et al. (2014) and te Brömmelstroet (2013), the usefulness of PSS can be measured by different aspects or dimensions. However, it is worth mentioning that PSS usefulness is often demonstrated and asserted in PSS experimental case studies (Champlin et al. 2019; Pelzer et al. 2016; Pelzer 2015). Results show that the benefits obtained in an experimental environment, to a large extent do not meet the actual needs of users and planning practices (Geertman 2017; te Brömmelstroet 2013; Vonk 2006). Studies also tend to emphasize the importance of improving PSS functionalities and usability in optimizing PSS usefulness (Pan and Deal 2020; Silva et al. 2017; Pelzer 2017). Thus, there has been much argument that PSS validity and potential would benefit from being employed in a context-rich, real-world research scenario, as well as much lobbying for such employment (Jiang et al. 2020a, 2020c; Luque-Martín and Pfeffer 2020; McEvoy et al. 2019; Zhang et al. 2019; Vonk and Geertman 2008), since it would provide PSS with “a structure [to] constructively develop a coherent and comprehensive body of knowledge” (te Brömmelstroet 2013: 306). In the present study, we conducted a survey to determine the multidimensional impact of contextual factors on the usefulness of PSS in smart city practices. Although PSS usefulness consists of different levels and dimensions, we applied the subjective perception of the overall satisfaction of the supportive role of PSS in practice—“overall usefulness”—to measure PSS usefulness (Jiang et al. 2020a).

Research Design

Data Collection

In order to study the effects of contextual factors on the usefulness of PSS in smart cities, we used an international questionnaire to gather and consolidate in-depth data and information concerning the application of PSS in smart city practices and to track performance across 55 quantitative indicators. The questionnaire was distributed to approximately 1300 members of the Computers in Urban Planning and Urban Management (CUPUM) research community across a wide geographical area. The CUPUM community was selected for two reasons. First, CUPUM is a major international academic platform that offers a state-of-the-art overview of the availability and applications of PSS-based methods, tools, and techniques, and it provides valuable and meticulous insights into a diverse range of social and environmental issues in the context of smart cities, big data, and smart urban futures (Geertman et al. 2019). Second, the perspectives of scholars and practitioners reflect the empirical case of applying PSS in practice, which provides a highly professional overview of the influence of contextual factors on PSS usefulness in the context of smart cities. The questionnaire, which was delivered by regular and electronic mailing systems, was administered over a 3-month period (May–July 2019).

The questionnaire consisted of three main parts. The first part gathered the (anonymous) participants’ basic information (e.g., gender, age, profession, origin, expertise with planning support ICTs), whereas the second part gathered in-depth information about their knowledge regarding the application of PSS in practical smart city projects. The respondents were specifically asked whether in recent years they had been academically and/or professionally involved in smart city projects. The follow-up questions chiefly assessed the performance level of five indicators—namely urban problems, functionalities, usability, added value, and context—perceived by researchers and PSS experts. The questions in the third part were related to specifics of the 2019 CUPUM conference and were not of relevance to this research.

Information about the influence of contextual factors on the usefulness of PSS in smart cities was obtained from the second part of the questionnaire. The development of the statements was based on previously recognized PSS literature. In order to effectively measure the strength of an individual’s perception of the importance and attainment of contextual factors’ roles, a 7-point rating system was used (1 = low, 7 = high). Respondents were not expected to respond to every statement.

Data Processing

This study used only the 18 statements related to the indicators presented in Section 2 to analyze the effects of context on PSS usefulness. It should be noted that “overall usefulness” (measured on a scale from 0 to 100 in the questionnaire) was recategorized into ordinal scale (from 1 to 5) to fit the relationship between the effects of contextual factors and PSS usefulness. “Types of urban issues” was regrouped as “economic issues,” “social–environmental issues,” and “mixed urban issues.” “Profession” was categorized as “academic researcher” and “practitioner.” The “academic research” group included academic researchers/scholars and doctoral students, while the “practitioner” group consisted of designers, politicians, and planners. As mentioned, “planning style” was categorized as “closed planning” and “interactive planning.” All other indicators were measured on a 7-point scale. The value assignment types for each indicator are shown in the second column of Table 1.

Table 1 Variable measurement and descriptive statistics
Full size table

Statistical Analysis

Of the approximately 1300 questionnaires that were distributed, 268 were completed (response rate of just over 20%). Of these, 175 were completed by respondents who had been involved in smart city projects in which the role of ICTs was significant, and we used their questionnaires in our analysis. In terms of geographical origin, more than half of the respondents were from China (53%); the other respondents were from Europe (15.4%), Asia (excluding China) (14.2%), North America (5.1%), South America (5.1%), Oceania (5.1%), and Africa (2.3%). Approximately 61.7% of the respondents were academic researchers/scholars (including doctoral students); the others (practitioners) were planners (32.6%), politicians (2.3%), or designers (3.4%).

Descriptive statistics were used to summarize the data (Table 1 and Appendix 1). Among all participants, the average score on “overall usefulness” was 2.90. Regarding “characteristics of the technology,” the average scores on “analyzing,” “designing,” and “transparency” are the top three indicators, whereas “communicating” had relatively lower scores (4.34). This indicates the different influences of the indicators of this variable. In terms of “characteristics of the urban issue,” the majority of the participants were involved in projects dealing with “mixed urban issues.” For the variable “user characteristics,” 61.7% of the participants were academic researchers, whereas 38.3% were practitioners, as mentioned. The average score on “expertise level” was much higher than the scores on “user attitude,” indicating the importance of “expertise level” in influencing PSS usefulness. Concerning “characteristics of the planning process,” the indicator “time pressure” received higher scores than “active uptaker” and “funding.” Further, 44.6% of the participants were engaged in “closed planning” and 55.4% in “interactive planning.” Finally, the average score on “political pressure” was 3.79, showing its relatively large influence.

Results

Model Fitting Information

To determine whether the effects of the argued contextual factors are statistically significant, ordinal logistical regression was employed to determine their associations. Table 2 shows the strength of associations, the predicted ability of the model, and goodness-of-fit statistics. Regarding the overall model test of the null hypothesis, the results yield a significance level of 0.000. Thus, it can be concluded that this is an important test, because the change in the likelihood function has a chi-square distribution (Harrell Jr 2015). The pseudo-R2 statistics measures the success of the model in explaining the variations in the data, which is an indication of the strength of the association between the dependent (overall usefulness) and the independent variables (contextual factors). Since the results of Cox and Snell (0.878), Nagelkerke (0.955), and McFadden (0.834) are smaller than those for a linear model, the pseudo-R2 can be regarded as very satisfactory (Harrell Jr 2015). However, because the number of empty cells in the model is relatively large (there are 456 cells (80.0% of the total number) with zero frequencies), the goodness-of-fit measures of Pearson and Deviance are not reliable.

Table 2 Overall model-fitting information, strength of association, goodness-of-fit-statisticsa
Full size table

Ordinal Regression Model to Explain PSS Usefulness

The detailed results of the calculations are summarized in Appendix 2. All variables were simultaneously added to the model along with a statistically significant two-way interaction term. The test of the proportional odds assumption was non-significant for the two models, suggesting that effects were proportional across the categories of the outcome variables. The estimates indicate that several variables representing “characteristics of the technology,” “user characteristics,” “characteristics of the planning process,” and “political context” have a strong influence on PSS usefulness. The variables that were determined as statistically significant are presented in Table 3.

Table 3 Parameter estimates determined as significant
Full size table

In particular, the regression coefficient for overall usefulness has a positive value and the observed significance of the “overall usefulness” is satisfactory (all p value≤0.05). The positive sign indicates that the influence of the contextual factors in general is positively related to the level of PSS usefulness. In other words, improvements in the positive effects and/or limitations on the negative impact of some contextual factors can increase the usefulness of PSS.

In terms of “characteristics of the technology,” some indicators show a positive sign whereas others are negatively related to PSS usefulness. First, the estimates of “informing,” “communicating,” and “transparency” are positive, which means the better performance of these indicators in practice leads to a higher evaluation of PSS usefulness. However, the positive relationship between these indicators and PSS usefulness varies. For instance, the influence of “informing” is only significant at level 2, whereas the significance of “communicating” is determined at levels 5 and 6. This reflects that the higher score on “communicating” could contribute significantly to the increase in PSS usefulness. Finally, the influence of “transparency” is significant at all 6 levels, indicating that the extent to which the underlying models and variables of the PSS are visible to users is important for PSS usefulness.

Second, “analyzing,” “designing,” and “flexibility” show negative signs, which means that a high score on these functionalities leads to lower usefulness perception. In terms of “analyzing” and “designing,” the result might seem odd and contradict findings presented in literature. A plausible explanation is that the frequency of the use of “analyzing” and “designing” in practice is high; thus, even low scores on these functionalities could lead to a high evaluation of the overall usefulness of PSS. Regarding “flexibility,” it is understandable that tools that can handle different planning tasks often indicates low specialization. In other words, users prefer more specialized tools.

As regards the variable “user characteristics,” the estimates of the indicator “user attitude” have a positive sign, indicating that users with a high positive attitude toward using PSS are more likely to give PSS a higher score on usefulness in practice. The estimate of “expertise level” is negatively associated with PSS usefulness at levels 2 and 3, indicating that a high level of expertise could lead to low PSS usefulness. This seemingly odd result perhaps arose because the majority of the participants were academic researchers and practitioners with high levels of expertise; therefore, they could have been more critical about the supportive role of PSS tools and thus gave a low evaluation of PSS usefulness.

In terms of “characteristics of the planning process,” the positive estimate of “active uptaker” implies that active users have a positive influence on PSS usefulness. As Vonk and Geertman (2008) argue, active users act as a catalyst because of their demonstration effect; that is, they encourage other stakeholders to emulate their behaviors via observation of the actions.

The estimate of the variable “planner style” shows a negative sign and the relevant p value is 0.002. This means that users in “closed planning” are more likely to give high scores to PSS usefulness. This can be understood from a reverse perspective, namely that closed planning processes are less influenced by those factors that appear in interactive planning, such as information overload, conflict of interest occurring when different actors oppose each other, and the difficulty of facilitating interpersonal communication and reaching consensus through PSS (Luque-Martín and Pfeffer 2020; Pelzer 2015; Vonk 2006).

The estimate of “political pressure” shows a negative sign, indicating that political pressure has a strong negative influence on PSS usefulness. According to Cairney (2016), political pressure and intervention highly affect the cognition, perception, and actions of the stakeholders involved in the policy-making process. As for PSS users in planning practice, politically motivated external pressure has a significant influence on the emotional and physical condition of users and how PSS is implemented and performed.

Finally, it should be noted that a range of other contextual factors (e.g., “types of urban problems” in Appendix 2) are not statistically significant, which means these indicators do not appear to be related to the level of PSS usefulness, at least for the current model and the study period.

Discussion

The results of the present study suggest an association between contextual factors and PSS usefulness, namely that, in general, “characteristics of the technology,” “user characteristics,” “characteristics of the planning process,” and “political context” have a significant influence on the usefulness of PSS, although their impacts vary significantly in practice. In particular, the indicators of “characteristics of the planning process” showed a positive sign, whereas the indicators of “political context” were found to be negatively associated with PSS usefulness. The indicators of “characteristics of the technology” and “user characteristics” showed both positive and negative associations with PSS usefulness. In addition, PSS usefulness can be better achieved in “closed planning.” Some of the results presented in this paper are in line with previous studies, while others are in contradiction with them.

First, the findings concerning the effects of usability indicators on PSS usefulness correspond with Pelzer’s argument that while “usability can be a necessary condition to achieve usefulness, it is never a sufficient condition” (Pelzer 2017:94). For instance, some studies found that simple and transparent PSS applications were positively associated with PSS usefulness (Pan and Deal 2020; Russo et al. 2018; Vonk and Ligtenberg 2010). The results obtained in terms of the “transparency” indicator confirm their findings. Silva et al. (2017) surveyed a number of instrument developers and found that user-friendliness improvement may make a limited contribution to the successful implementation of accessibility concepts in planning practice. In the present study, the influence of “user friendliness” on PSS usefulness was not significant. However, it should be noted that in an experimental environment, user-friendliness is significant in terms of its influence on PSS usefulness (Pelzer 2017). A possible explanation for this discrepancy is that the broader contextual factors considered in this study downplayed the significance of user-friendliness.

In terms of the functionality indicators, previous studies found that PSS that are able to facilitate social interaction, interpersonal communication, and community debate can empower professionals and citizens alike to have a better PSS experience and make better informed decisions (Zhang et al. 2019; Saad-Sulonen 2012; Pelzer 2015). Our findings indicate that the influence of “informing” and “communicating” functionalities is statistically positively associated with the overall evaluation of PSS usefulness, revealing its positive effects. Nevertheless, the findings concerning “analyzing” and “designing” functionalities contradict previous findings. For instance, Moghadam and Lombardi (2019), Vonk and Ligtenberg (2010), and Pettit (2005) found that analyzing and designing tools allows decision makers to systematically generate and evaluate alternative solutions, gaining insights that were introduced as inputs to help guide further analyses (e.g., GIS, Maptable, What-If and CommunityViz). However, the large-scale survey carried out for the present study showed that the significance of “analyzing” and “designing” functionalities is negative rather than positive. As mentioned, the high implementation frequency of these tools is a plausible reason for this.

Second, the findings in terms of the variable “user characteristics” correspond with those of earlier studies on how users define technological functions and their use. As Geertman and Stillwell (2004) argue, the user interface of a PSS should be sensitive to the characteristics of the user and to the kind of information that it communicates to that user. The findings of the present study confirm their argument that a high level of positive attitude toward using PSS would improve the awareness of the benefits of using PSS. This finding supports Vonk (2006), who found that a low level of intention to use PSS among possible users reduces the potential expectation of PSS usefulness. Nevertheless, the results obtained concerning “user characteristics” also seem to contradict earlier results. For instance, Holsapple et al. (2005) and Hoc et al. (2013) show that higher educated users have greater user satisfaction due to their ability to learn and to develop a high level of expertise. However, the findings presented in this paper show that this relationship could be negatively associated, since experts with a high level of expertise tend to have a fault-finding attitude. As mentioned, they are often more critical of the performance of PSS tools in practice.

Third, the negative influence of “active uptaker” shows that in order to improve awareness of the existence and potential of PSS, more effort should be made to demonstrate to users the benefits of applying the diversity of PSS (Vonk and Geertman 2008). Only such an effort will increase the awareness of PSS, making the PSS message more likely to be picked up and appreciated by employees of planning organizations (Vonk et al. 2005).

Fourth, it is interesting that “closed planning” seems to be more beneficial for optimizing the supportive role of PSS. Although few previous studies have compared the performance of PSS in these two distinctive planning styles (Vonk and Ligtenberg 2010; Pelzer et al. 2015), an inverse perspective can be used to explain this finding. For example, in ICT-facilitated interactive planning, PSS are more vulnerable to externalities and a range of side effects or consequences.

Fifth, the significant negative influence of “political context” on PSS usefulness confirms the claims by some authors that a broad field of politics impacts the likelihood that actors in the process will absorb new PSS knowledge, and that they will be able to use this knowledge to assess problems and find solutions. For instance, Zhang et al. (2019) identified that in the earlier divergence period, the top-down government-led approach in China often limited the usefulness of web-based PSS in eliciting ideas from independent citizens and supporting participation and engagement. And te Brömmelstroet (2015) shows that in the later convergence period, power relations and hierarchical structures affected the group dynamics and how PSS perform.

Finally, we hypothesized that “characteristics of the urban issue” have a significant influence on PSS usefulness. Some studies highlight that the attributes of the urban issue help judge the appropriateness of the supportive role (Jiang et al. 2020c; Pelzer et al. 2015; Vonk and Ligtenberg 2010), and thus their importance should be given more attention. However, the estimate of the indicator “types of urban problems” was not found to be as significant due to the broader multiple and multidimensional aspects of context impact.

The present study had some limitations. Because of our selection procedure, the opinions and attitudes of civil society were not considered. According to some authors, however, ideas and knowledge from civil society can effectively promote the advancement of PSS (Geertman and Stillwell 2020; Zhang et al. 2019; Pelzer 2015). It should be noted that although the usefulness of PSS is comprised of different dimensions, in this study we considered only the overall evaluation of PSS usefulness. The importance of these different kinds of usefulness of PSS was acknowledged in another article by Jiang et al. (2020a). Thirdly, it should be noted that not all the functionality and usability indicators were discussed in this paper due to the semantic overlap between some concepts (e.g., communicative support capabilities as part of functionality, and communicative value as a usability indicator) (Pelzer 2017).

Conclusion

Studies on PSS usefulness pay much attention to developing a conceptual and empirical understanding of the relation between planning tasks and PSS (i.e., task–technology fit) and the interaction between the user and a PSS (i.e., human–computer interaction) (Pan and Deal 2020; Jiang et al. 2020a; Russo et al. 2018; Silva et al. 2017; Pelzer 2017; Pelzer et al. 2014; te Brömmelstroet 2012). As a consequence, there is a lack of empirical evidence concerning the importance of the multidimensional aspects of contextual factors in analyzing the development, implementation, and effects of PSS in planning practice. In the present study, we comprehensively examined the extent to which contextual factors influence PSS usefulness via an international questionnaire survey. This paper contributes empirically to insights that “the way in which [PSS] are handled within a specific planning situation will enhance their potential for performing a more effective planning-supportive role (Geertman 2006: 878).” While this study did not provide evidence that all contextual factors are associated with PSS usefulness, our results suggest that some contextual factors (i.e., “characteristics of the technology,” “user characteristics,” “characteristics of the planning process,” and “political context”) indeed have differentiated effects on the usefulness of PSS. The results of our study expand the knowledge of the complex relations between contextual factors and PSS usefulness in the realm of smart cities. Base on this, we emphasize that only when PSS users can identify the critical contextual factors that are favorable and unfavorable, will the potential benefits of PSS for spatial planning be fully achieved.