1 Introduction

The architecture, engineering, and construction (AEC) industry plays a key role in how our world will look in the future. It is facing the challenge that buildings, housing, and infrastructure for more than 2.5 billion people will have to be realized in the next decades. An important part of improving AEC is to advance the way architecture is designed and optimized to reduce energy use, greenhouse gas emissions, and waste. However, despite their great potential, computational optimization methods have had a limited impact on architectural design so far [33, 44, 51], especially when compared with other engineering domains where the use of optimization is common practice [14].

Optimization inherently requires a well-defined problem with clear constraints and objective functions. While such elements might be available at a late stage of the design process, it is rarely the case at early-stage design phases, where design is often described as a wicked problem with ill-defined tasks and incomplete data [1, 10, 44]. At this stage, designers have much freedom but little knowledge about the problem at hand [33]. Further, when exploring optimization spaces designers are not only interested in finding the optimal solution but rather in exploring a wide variety of solutions that obtain an acceptable performance and in understanding how the different design parameters interact with each other and influence the performance. In their interview with 18 practitioners from the AEC sector, Bradner et al. [4] found that “... the computed optimum was often used as the starting point for design exploration, not the end product” and that designers need to “... describe the solution and design spaces.”

These characteristics call for human-in-the-loop-based solutions rather than fully automatic ones, which is an essential prerequisite for visual analytics approaches. In particular, techniques for parameter space analysis and exploration offer a great opportunity not only to find an optimal design for a given set of parameters but also to help designers gain a deeper understanding of the design space.

However, designing and building these solutions comes with challenges. In the visualization community, Sedlmair et al. [38] surveyed 21 tools for visual parameter space analysis. The authors identified the limited sampling capabilities (i.e., data acquisition gap), the challenge of building good surrogate models (i.e., data analysis gap), and the issue of visually navigating the multidimensional spaces (i.e., data cognition gap) as the three main challenges when it comes to building these tools. These challenges are amplified in the AEC industry due to the lack of digitization [1, 27]. Most of the existing parametric toolboxes and optimization solvers [5, 6, 43] are limited when it comes to navigating the multidimensional spaces of the optimization (i.e., data cognition gap) and often restricting when it comes to discovering new solutions (i.e., data analysis and acquisition gaps). An exception are performance maps [48, 49], where the authors address the data cognition gap using star coordinates to obtain a 2D projection of the optimization space. Zorn [52] built on top of that concept by exploring the role of sampling methods and surrogate models and how they influence the resulting landscape. While performance maps provide a continuous overview of the optimization space, they are lacking when it comes to revealing the relationships between the design parameters and the objective function. Additionally, star coordinates are linear projections that do not account for local neighborhoods in the data.

Our work builds on and extends previous work [48, 49, 52] by introducing a visual analytics framework (Fig. 1) to explore fitness landscapes of a single-objective optimization function. The main contributions of our work are as follows: (1) A visual analytics framework for exploring fitness landscapes that span across data, projection, and visualization layers. The framework is embedded in an existing workflow for architectural design. (2) A parameter study to investigate the role of surrogate models, sampling strategies, and number of samples in constructing fitness landscapes. (3) A user study involving 12 domain experts to examine user preferences, navigation strategies, and evaluate understandability and interpretability of the visualization and projection layers within our framework. (4) A case study with an expert and data from the architecture domain.

In contrast to other application areas, the AEC domain remains relatively untapped and has yet to harness the full power of visualization and visual analytics [1, 27]. Our paper is a result of a close collaboration with experts from the architecture domain over the span of nine months. We see our work as a step to bridge the gap between the architecture and the visualization communities, providing insights into how visualization can support computational design and the analysis of complex parameter spaces.

Fig. 1
figure 1

Visual analytics framework for architectural design optimization. The main view offers two alternative representations for exploring the design space: a continuous density map and a discrete glyph-based map. Objective values and parameter view are updated based on the cursor position. The glyph-based map features a portal cursor for semantic zoom, displaying data in a separate portal lens view. 3D renderings of the underlying structure are displayed for selected parameter constellations. Further, the exploration history shows which areas of the landscape have been investigated

Full size image

2 Related work

Related work comprises computational optimization in the architectural domain and, in general, visualization, exploration, and analysis of multivariate data.

2.1 Architectural design optimization

The architecture domain is not short of optimization solvers [42, 43, 48] or simulation tools [24, 28]. However, most of these tools, while computationally powerful, have limited visualization capabilities when it comes to exploring the resulting optimization and simulation spaces. More than a decade ago, Erhan et al. [12] introduced ViSA, a visual sensitivity analysis approach for parametric modeling. They reached out across domains and argued in favor of visual analytics and the consideration of human cognitive systems when building such tools. StructureFIT [33] and EvoMass [44, 45] are tools for exploring different design options resulting from evolutionary algorithms. Design explorer [5, 6] and Dream Lens [30] are toolkits that offer different sampling strategies with a special focus on design variables analysis and transformation (i.e., sensitivity analysis). While these tools support design space exploration, they rely on design catalogs for presenting the varying design options and/or overlaying the information directly on the 3D structures. We aim to utilize data visualization to gain a high-level understanding of the design space and the nature of the optimization problem.

There are few prior works that utilized abstract data visualizations for exploring optimization spaces. Asl et al. [3] introduced a visual programming interface combined with parallel coordinates for exploring design alternatives resulting from multi-objective optimization. Fuchkina et al. [15] introduced a framework for exploring parametric modeling with node-link diagrams, parallel coordinates plots, and self-organizing maps. In the context of the fitness landscape, Chen at al. [7] and Stasiuk et al. [41] used k-means clustering to find relationships between the design parameters and the objective function and visualized one representative of each cluster. We distinguish ourselves from this previous work in three ways: (1) We focus on single-objective optimization problems. (2) We aim to obtain a direct overview of the fitness landscape that shows the relationships between the design parameters and the objective function rather than inferring these relationships by a few representative samples. (3) We use star coordinates and dimensionality reduction methods to obtain a 2D spatial mapping of the design space. We position ourselves close to the work of Wortmann et al. [48, 49] and Zorn [52], who employed star coordinates and surrogate models to obtain a continuous overview of the fitness landscape. In comparison, we complement the continuous overview with a glyph-based visualization to reveal the relationships between the design parameters and the objective function. Likewise, we complement the star coordinates with UMAP to preserve local neighborhoods in the data. Additionally, we investigate the influence of the number of samples on the resulting fitness landscapes. Furthermore, we carry out an empirical user evaluation and present a case study, which has not been done by previous work.

2.2 Visualization of multivariate data

Despite recent advances [29], parallel coordinates plots (PCPs) [18, 21] and scatterplot matrices (SPLOMs) [2] remain the two most prominent ways for visualizing multivariate or multidimensional data. SPLOMs show pair-wise relationships between the data variables in scatterplots and are familiar to most users. On the downside, relationships between more than two variables are hard to perceive, and the space consumed by the plot grows quadratically with the number of variables. PCPs represent data items as polylines through equidistant vertical axes. In contrast to SPLOMs, the space consumed by PCPs grows proportionally with the number of variables. While PCPs, in principle, are capable of showing the relationships between the variables, they rely on axes ordering [46] and are prone to visual clutter with an increasing number of data items and therefore need clutter reduction methods [11, 34].

To mitigate these problems and obtain a spatial overview, which is missing in both techniques, radial-based projection techniques such as RadViz [20] and star coordinates [23] depict each data item as a point in a 2D radial layout. While the two approaches project the data in a similar way, RadViz introduces a nonlinear normalization step that hinders outlier detection and makes it challenging to find a useful linear mapping of the resulting plot [36]. Star coordinates, in contrast, preserve linearity in the data and, with proper axes calibration, can recover the attributes of the projected data items [37]. Dimensionality reduction (DR) offers an alternative way to obtain a 2D spatial overview of high-dimensional data. Espadoto et al. [13] quantitatively surveyed 44 of these methods across 8 different traits, such as linearity, preservation of neighborhood, complexity, and inverse-transformation ability. Radial projections and DR methods are known to be challenging when it comes to interpreting the projection results. Several techniques have been proposed to mitigate this problem—by showing projection errors [40], displaying data items in the context of the original dimensions [8], or using glyphs [16] as a unit visualization for representing the points in 2D spaces [22, 25].

We position the visualization aspect of our work close to these techniques. On the one hand, we make use of star coordinates and DR methods to obtain an overview of the optimization space. On the other hand, by incorporating glyphs into this overview, we improve the interpretability of the projection results and depict the relationships between the design parameters and the corresponding objective function, which is a desired functionality when exploring optimization spaces. In our work, we took inspiration from Kindlmann [26] and Hlawatsch et al. [19], who used geometric primitives and radar glyphs to visualize a continuous field. In our case, while we share the continuity aspect of the data, we use star glyphs to reach the same goal.

3 Methodology and requirements

We followed the design study methodology [39] to elicit the requirements, design, build, and evaluate our framework. On an abstract level, the full cycle (from requirement analysis to evaluation) was completed twice. The focus of the first cycle was mainly on the visualization layer, whereas the second cycle addressed the data and projection layers. Our requirement elicitation phase drew from the findings obtained by Bradner et al. [4] and Wortmann et al. [50]. These findings were further refined through a series of semi-structured interviews with three experts in visualization, two experts in architectural design optimization, and one software developer; all are co-authors of this manuscript. Each requirement phase was followed by design, implementation, and evaluation phases. The different phases often overlapped, and the process was highly iterative. In total, the project spanned over nine months.

Architectural design often aims to optimize the design parameters with respect to one or more objective functions (i.e., performance criteria). These performance criteria could be, for example, energy usage, daylight exposure, or material usage. The multidimensional design space combined with the objective function(s), when visualized, is often referred to as a fitness landscape or a performance map [49]. We identified the following requirements for building and designing tools to visualize fitness landscapes.

R1: A diverse landscape Although finding the optimum solution is a fundamental requirement, it can be fulfilled without a visualization tool since the optimum is a direct result of the optimization algorithm. Designers often seek not only the optimum but also a multitude of design alternatives that obtain an adequate performance. A diverse landscape with a wide array of solutions is usually highly valued by designers.

R2: Spatial mapping of the design space While the design space can be visualized using various multivariate data visualization methods such as PCPs or SPLOMs, designers tend to think about high-dimensional space in terms of 2D or 3D spatial layout [47, 49]. A spatial representation enhances designers’ ability to identify regions of interest, clusters of similar solutions, and areas of the design space that may be under or over-sampled.

R3: Understanding the design space Uncovering relationships between the design parameters and the performance is an important step toward understanding the design space. Knowing which parameters have a significant impact on the performance is as important as knowing those that have no impact. The former poses a restriction, whereas the latter gives more design freedom. Similarly, uncovering correlations and potential trade-offs among the parameters themselves leads to a deeper understanding of the design space.

R4: 3D structure Having the ability to view the corresponding 3D structure of a solution is a key feature that the visual analytics framework should provide. This is important since not every well-performing solution is a feasible design. Examining the 3D structure represents the final step for determining a solution’s feasibility.

R5: Interactivity and workflow integration The ability to navigate the design space, compare different solutions, and examine them in a seamless way is highly desirable. Moreover, it is crucial to integrate the visual analytics approach into existing design workflows for better adoption and usability. We provide compatibility with Rhino/Grasshopper as it is the most common platform [50] for computational design and optimization and the one used by our domain experts.

4 Visual analytics framework

We designed a visual analytics framework to fulfill the aforementioned requirements. As displayed in Fig. 2, it comprises a data, a projection, and a visualization layer. The data layer includes surrogate models to provide efficient design solutions and sampling strategies for the multivariate data parameters. The projection layer provides different spatial layouts for the data results and the visualization layer displays the resulting fitness landscapes with continuous and glyph-based representations of the data. The user can interact with all three layers to explore the parameter space.

Fig. 2
figure 2

Our visual analytics approach comprises three layers, i.e., data, projection, and visualization layers. Domain experts can interact with each layer to adjust simulation data, projection methods, and explore the data with different visualizations

Full size image

4.1 Data and parameter study

To fulfill requirement R1, we adopt two surrogate models and three sampling strategies introduced by Zorn [52] to obtain a wider array of landscapes. We first introduce the surrogate models and sampling methods, followed by a brief discussion of our findings from the parameter study.

Surrogate models Surrogate models construct a mathematical approximation of the design space based on the optimization parameters and use such a model to predict unknown sample points. Therefore, they are computationally cheap compared to running physics simulations that might take a few hours or days [51]. In our work, we leverage surrogate models by experimenting with (1) a radial basis function interpolation surrogate model trained using RBFOpt optimizer [9] and (2) a surrogate model resulting from a gradient boosted regression tree (GBRT) model trained on a varying number of samples selected using the Latin hypercube sampling strategy. While (1) optimizes its hyperparameters during the optimization process, we use a Bayesian optimization algorithm to optimize the hyperparameters of (2).

Sampling strategies We investigate three common sampling strategies: (1) Optimization-based sampling, (2) Latin hypercube (LH) sampling, (3) hybrid sampling strategy that combines both of the above. Since expensive physics simulations result in long run times, it is unfeasible to sample more than a couple of hundred samples before exploring the design space. Therefore, we limit the number of samples for evaluating the framework to three sets of 100, 500, and 1000 samples, respectively.

Parameter study To investigate the influence of the surrogate model, the sampling strategy, and the number of samples on the resulting fitness landscapes, we conducted a parameter study based on two datasets. The study showed that RBF landscapes with optimization-based samples, while giving us some degree of global overview (exploration), many points are clustered in areas of high performance (exploitation) (see Fig. 3a). Thus, the resulting landscape does not provide a holistic overview of the design space. On the contrary, sampling a multi-dimensional space in an almost random fashion, as in the case of GRBT landscapes with LH samples, is less likely to obtain many high-performing solutions (see Fig. 3b). RBF, when combined with hybrid samples, leads to a diverse landscape (see Fig. 3c) with solutions ranging from high-performing on the left of the map to low-performing ones on the right. Such landscapes might be valuable for designers. For more details on the parameter study and the complete set of results, we refer to the supplemental materials.

Fig. 3
figure 3

RBF and GBRT surrogate models trained on 1000 samples with respect to three sampling strategies. RBF with hybrid samples produces a diverse landscape combining low and high-performant solutions

Full size image

4.2 Projection

Projecting the sample points to a 2D plane allows us to obtain an overview of the fitness landscape, which is essential for exploration (R2). We use star coordinates (SC) and uniform manifold approximation and projection (UMAP) [31] to obtain two alternative projections of the fitness landscape. SC is easy to navigate if one is looking to retrieve a solution with a high value at a single or two adjacent parameters on the coordinate system. These solutions are usually located at the outer boundaries of the map in the direction of the corresponding parameter. However, as we move toward the center of the map, it becomes more ambiguous which parameters contributed the most to a given solution since two solutions of different parameter values could be projected close to each other. UMAP, in contrast, preserves local neighborhoods in the data and, therefore, promises to reveal different clusters of similar solutions. Nevertheless, it comes at the cost of being less intuitive to interpret compared to SC, especially without a layout enrichment strategy [22, 40].

4.3 Visualization

To better understand the design space (R3), we provide two alternative representations for fitness landscapes: continuous and glyph-based. The continuous representation offers a more precise depiction of the optimization space, whereas the glyph-based is better at revealing the relationships between the design parameters and the performance value.

Continuous fitness landscape (CFL) To render a CFL, we estimate the performance values to fill in the missing data between the projected sample points. Interpolating the performance values directly is fast but provides no information beyond the sample points because it does not use the underlying model. Instead, we perform interpolation in the parameter space followed by a query to the surrogate model to get the corresponding performance value. We investigate two different methods: (1) Barycentric interpolation relies on a triangulation of the evaluated and projected points, whereas (2) Shepard interpolation takes all projected points into account, weighting them less with increasing distance. Figure 4 shows an example of the two interpolation methods. With a power parameter of \(p=2\) for the Shepard interpolation, we also take points with higher distances into account.

Fig. 4
figure 4

Different interpolation methods to represent a CFL: a Barycentric interpolation triangulates evaluated and projected points, b Shepard interpolation considers all points, giving less weight with increasing distance

Full size image

Glyph-based fitness landscape (GFL) CFLs provide an overview of the optimization space but lack the ability to depict the relationships between the design parameters and performance (R3). Hence, we complement the CFL with a glyph-based fitness landscape to fill this gap. The decision to opt for a glyph-based design was motivated by the low number of attributes featured in architectural design optimization problems [50]. While the glyph design space is huge [16], we decided on star glyph as they are easy to understand and fit with the SC projection. Furthermore, star glyphs are very good at forming recognizable shapes and patterns [25], which, in our case, is crucial to maintain the sense of continuity in the fitness landscape. Figure 5a shows a star glyph representing one data item of ten parameters. The corresponding objective value of each data item is double-encoded by the color of the formed star shape and the background color of the glyph. We use a high opacity value for the background color to distinguish the actual values from the maximum values at each parameter. Utilizing the background color allows us to highlight the small parameter values and to visually sustain the sense of continuity when glyphs are packed in the GFL (see Fig. 5b). To construct the entire GFL, we uniformly discretize the CFL. The discretization rate is a user-defined parameter and implicitly defines the glyph size. At lower discretization rates (i.e., big glyph sizes), more data items are aggregated into a single glyph. Depending on the sampling strategy and the projection technique employed, these data items might vary in the parameter values and/or the corresponding objective value. To account for such variability, we utilize the glyph stroke to encode the standard deviation within the glyph.

Fig. 5
figure 5

a Star glyph for multivariate input data and a respective objective value. The color encodes the objective value and the outline of the glyph helps estimate parameter ranges and the diversity of the underlying data. b The GFL visualizes the relationship between the design parameter and the objective value

Full size image

Using glyphs allows us to visualize the design parameters, encoded by the glyphs, in relation to a single-objective value, encoded by color. The fitness landscape relies on DR methods to obtain an overview of the optimization space. Interpreting such an overview might not be an easy task due to the projection artifacts. Therefore, using glyphs also serves to improve the interpretability of the projection results. While using glyphs to visualize the projection results is not new [22, 25], the novelty is using them to visualize CFLs and extending the design to show multiple levels of detail.

4.4 Interactive framework

For a framework showing fitness landscapes and 3D structures (R4) integrated into the design workflow (R5) of architects, we implemented a visual analytics approach consisting of multiple views that provide an overview and details for different parameter constellations and their respective 3D representation (Fig. 1). The main view at the center can be changed from a continuous view to a grid view, showing the CFL or GFL, respectively. Both representations, as well as different interpolation methods and color scales, can be changed on demand.

The objective value and the parameter view update live with the mouse movement in the main view, providing parameters and objective values in numbers for the currently hovered glyph or point on the map. Similarly, the render view displays the 3D representation when clicking a sample point. When GFL is selected, the user has the option to adjust the resolution level of the grid.

To support the exploration process, we adapted the idea of a minimap that shows the exploration history, indicating where analysts have already inspected the visualization with the cursor. With this visual aid, it is possible to track the exploration process and focus, for instance, on areas that were not investigated in detail. To provide the full spatial context of the exploration history, it is also possible to show the generated density map superimposed on the main view.

Data drill-down In contrast to CFL, GFL requires more screen space to convey the additional information of parameter sets besides the objective value. Glyphs with thick strokes signify a high standard deviation and is typically an indicator to investigate a glyph in more detail. To explore the individual sub-glyphs at different zoom levels, we included two options: (1) An external view for the portal lens (Fig. 1). (2) An in-place portal lens that is displayed directly at the cursor position (Fig. 6). The portal lens is updated by moving the portal cursor on the grid, and the mouse wheel is used to change the semantic zoom level, which is visually indicated by a partial circle around the portal cursor in both options. The external view has the advantage of showing the glyphs in a larger view without occluding the overview. The in-place variant provides a data drill-down without splitting attention between two views on the screen. We implemented both options and found that users less familiar with performance maps preferred the in-place lens. With this design, it is possible to keep an overview of the fitness landscape while drilling down into specific solutions generated by the model.

Fig. 6
figure 6

Portal lens serving as a semantic zoom to investigate glyphs. Glyphs with a high deviation are split inside the lens, showing potential border regions where parameters changed

Full size image

5 Case study: optimization of coreless filament wound structures

To showcase our approach with real data in collaboration with a domain expert, we present a case study in the context of coreless filament winding (CFW). For detailed information on the procedure, we refer to the supplemental material.

5.1 Coreless filament wound structures

CFW is a manufacturing process that enables the fabrication of highly differentiated, lightweight, resource-effective, and high-performing building components [17]. In CFW, free-spanning fiber rovings are sequentially placed between anchor points, and the fiber net emerges from the fiber–fiber interactions [32], which is the optimization objective to maximize. The parameter space defines the boundary condition, i.e., the placement of the winding anchors and hyperparameters of the winding sequence. Designers want to explore the CFW component design space qualitatively, considering shapes and boundary conditions’ influence on the resulting fiber net. Depending on the amount of fiber interactions, simulation time varies from seconds to minutes.

5.2 Analysis insights

The expert analyzed the RBF map with optimization-based samples. During the analysis, the expert confirmed their hypotheses about the data but also discovered new insights.

5.2.1 Confirming hypothses

High anchor count correlates with higher objectives The domain expert confirmed such a hypothesis by investigating various solutions on the map. The expert commented that “the map looks a bit similar to what I expected... the solutions with high anchor count [one of the parameters] also have a higher objective.” This finding highlighted the importance of this parameter in the optimization process, indicating its direct impact on the quality of solutions.

The shift parameter influences surface curvature After rendering low and high-performing solutions, the domain expert noted a relationship between the shift parameter and the surface curvature of the structure. Lower shift values led to almost straight spanning components, indicating a flatter structure (Fig. 7c), while higher shift resulted in structures with more curvature due to increased fiber interactions (Fig. 7a, b). The expert added “this is also something we observed in the projects we are working with.”

Fig. 7
figure 7

CFW structures with triangular side frames and high curvature (a and b) result in a higher number of intersections (the red dots) than straight-spanning structures with circular side frames (c)

Full size image

5.2.2 New insights

Radius1, Radius2, and the Performance While using CFL, the expert noticed that the two parameters, radius1 and radius2, fluctuate significantly in the high-performance area. “What I would be curious to know is if there is a relationship between the two,” the expert added. To answer the question, the expert switched back to GFL and noticed that the two parameters must be of similar value to get a high-performing solution. This was considered one of the main insights they obtained during the analysis.

High-performing solutions often feature triangular frames After investigating the polygon side parameter, the expert was surprised that most high-performing solutions have a low polygon side. With this, the two frames of the structure appear more triangular, as in Fig. 7a or b, than circular, as in Fig. 7c. This was the second insight of the analysis. Understanding this relationship between parameters and the geometry of the solutions provides valuable information for designing structures with specific characteristics tailoring the optimization process to achieve desired outcomes.

5.2.3 Observations

Navigating the map The domain expert had no problems interpreting the SC. They had a very good understanding of where to move in the space, looking for certain parameter configurations. They also noted that the high-performing solutions were clustered on the left side of the map with many samples, while the low-performing ones were on the right side with few samples. They commented: “I guess that makes sense because the RBF tries to maximize.”

Visualization The expert appreciated both the CFL and GFL representations: “Actually, what I like more about the glyphs is that my focus always stays in this area [referring to the canvas area]... here [referring to the CFL] it is clearer to see where the samples are.” The expert frequently used the render view to examine the 3D representation of the structures. Additionally, they used the portal to investigate glyphs with thick strokes, indicating a high standard deviation. The expert also favored rendering the CFL using the Shepard interpolation over the barycentric method, citing that the former provided a more fluid and organized visual representation.

6 User study

We aimed to evaluate our approach with the main research question: “How do domain experts explore fitness landscapes under different projections and visualizations?” We conducted a qualitative experiment to probe the different combinations and identify differences in interpretability, task difficulties, applied strategies, and usability.

6.1 Study design and tasks

There are two variables to investigate in our study: the projection (SC vs. UMAP) and the visualization (CFL vs. GFL). Due to the limited number of domain experts, we chose a mixed study design. The goal here was to explore the design alternatives under qualitative aspects rather than by confirmatory, hypothesis-driven design. To probe the design space with respect to these variables, participants were exposed to either both projection methods and both visualization techniques (G1), both projection methods with only one visualization technique (G2), or both visualization techniques with only one projection method (G3). This way, we could probe overall impressions (G1), and more comparison-based feedback for projections (G2) and the visualizations (G3). The evaluation was based on the following tasks:

T1: Projection mental map Participants are provided with two parameters and are asked to approximate the location of a solution on the map, considering parameters having either low or high values.

T2: Design alternatives Participants have to find two or three design alternatives that achieve relatively high performance but differ in their parameter configurations.

T3: Significant parameters We ask participants to identify parameters that have a significant impact on the performance and identify the impact type (positive or negative).

T4: Parameters correlations Participants have to identify correlations between parameter pairs and the performance. Similar to task T3. They are also asked to identify the type of correlation.

Task T1 was designed to assess participants’ ability to comprehend and navigate the fitness landscape under the different projection methods. Considering its fundamental role in solving subsequent tasks, it was placed at the beginning of the evaluation and repeated twice. We refer to the two trials as T1.1 and T1.2. The selection of tasks T2–T4 was motivated by understanding the design space (R3).

Fig. 8
figure 8

Subjective rating of the task difficulty among 12 participants for the four tasks

Full size image

6.2 Procedure

Each task started with a brief introduction and ended with a difficulty rating on a 5-point Likert scale, a description of applied strategies, and preferred projection and/or visualization. In the end, we held an open discussion where we asked participants to rate understandability, interpretability, faithfulness, and trust of the projection method and/or visualization techniques they were exposed to. In total, we recruited \(N = 12\) participants (four identified as females) equally divided into three groups (i.e., four per group). All had a university degree in architecture (six master’s, five bachelor’s, and one Ph.D.) and eight indicated some familiarity with the Performance Map technique. The study received approval from the ethics committee of the university and participants were compensated at the rate of EUR 12 per hour. For more details regarding the study procedure, questionnaires, dataset, and stimuli, we refer to the supplemental material. In the following, we discuss the results.

Fig. 9
figure 9

Preferences of participants for a the preferred projection per task, b the preferred visualization per task. For the assigned groups G1–G3, we further evaluated c the preferred projection per visualization and d the preferred visualization per projection

Full size image

6.3 Tasks difficulty and preferences

Figure 8 shows the subjective rating of participants regarding the difficulty of each task. Task T3 stands out as the most difficult task. T4 was rated less difficult; this could be due to some learning effects. Tasks T1 and T2 were mostly rated easy to solve. Figure 9 shows the projection and visualization preferences of the participants. For the projection, we only considered the data from the study groups G1 and G2 since those participants had a choice to switch between the two projection methods. Likewise, for the visualization, we only considered the data from G1 and G3.

SC versus UMAP SC was the preferred projection method across all tasks. Participants used the words “easy to understand,” “easy to locate solutions,” “quick to learn,” and “straightforward” when asked to explain their preference. They also often referred to the axes of SC as the key that made their job a bit easier. UMAP did not have such axes in comparison, and therefore, it was harder to navigate the landscape. One participant stated: “I felt that CFL-UMAP showed more solutions, but I had to go hunting for them because I did not know where to start my search.” Participants favoring UMAP over SC mentioned reasons that were not necessarily related to how they read or understood the projection. Instead, they sometimes confused the influence of the projection method with the influence of the visualization method (i.e., when changing from GFL-SC to CFL-UMAP).

One participant, when asked about why they favored UAMP in task T1 answered: “In this combination [referring to CFL-UMAP] I had quick success.” They elaborated that the choice of projection method did not impact their decision: “I was ignoring what is happening there [referring to the canvas]... that was only a loose connection.” One participant favored UMAP in task T2 as it showed them more clusters than SC. They said: “SC was more confusing as only one cluster was visible. In UMAP, I can see three or four clusters.” The option “not sure, both” was frequently chosen by participants after attempting to solve tasks T3 and T4 (see Fig. 9a). Participants often switched between the projection methods to try to find an answer. One participant, however, chose this option in task T1 since SC showed them a solution in a location that did not align with their understanding of how SC works. This participant stated that “axis was confusing, even though I knew what it meant ... would rather not have an axis.” Fig. 9c shows the preferred projection grouped by visualization. On most occasions (13/15), when participants were exposed exclusively to CFL (group G2) or showed a preference for CFL over GFL when exposed to both (group G1), they also tended to favor SC over UMAP.

CFL versus GFL Figure 9b shows the participants’ preferences regarding the visualization technique. GFL and CFL were rated equally in tasks T1 and T2. In T3, while CFL was favored over GFL, participants were undecided. This, similar to the projection results (Fig. 9a), reflects the difficulty of T3, where participants had to switch between the two representations. T4, in contrast, clearly indicates a preference for GFL. When participants preferred CFL over GFL, they often pointed out that CFL was better in terms of distinguishing “color gradient” or “color brightness” compared to GFL. In contrast, participants who favored GFL often highlighted their ability to distinguish and compare “shapes” or “geometry” of different solutions, especially in T3 and T4.

Several participants came to the conclusion that both representations have their strengths. While CFL is better suited for exploratory analysis, GFL performs better regarding detailed investigation. One participant commented: “CFL more exploratory but GFL is more exact.” Another participant added: “to play around to find a spot, the first one [CFL] is easier to use. To have more information, it seems like the other one is more beneficial.” This participant also drew an analogy, that looking for a high-performing solution in CFL is like playing a game of blindfolded candy hunting. They added: “I just need to find one... but with a huge lack of certainty.” Fig. 9d shows the visualization preferences grouped by projection method. While the results suggest a tendency toward GFL when paired with UMAP, further investigation is needed to conclusively confirm such a hypothesis. Participants favored GFL over CFL, especially with barycentric interpolation. It produces triangular artifacts (Fig. 4), which made it less appealing when paired with CFL. Some participants expressed that these artifacts made them trust the visualization less.

6.4 Navigation strategies

Participants explored the maps by moving the mouse cursor on the main canvas and observing the changes in the other views. During the study, we observed three navigation patterns. These patterns were later confirmed by analyzing the heatmaps of the participants’ mouse trajectories. For more details, see the supplemental materials.

Moving a long axes Some of the participants who were exposed to SC projection started the exploration by moving in a circular motion around the SC axes. Once they found something interesting, they stopped and started moving up and down along the axis of interest. This strategy was often used when trying to solve tasks T3 and T4. This also can be seen in the heatmap of the mouse trajectories of one of the participants as in Fig. 10a.

Random search Randomly exploring the map was often done by participants when UMAP was used as the underlying projection. Nevertheless, other participants followed this strategy with SC when their reasoning about SC failed to deliver an answer (see Fig. 10b).

Systematic search Few participants preferred to scan column-by-column or row-by-row in a systematic manner (see Fig. 10c). While this behavior was more pronounced with GFL due to the grid layout, it was also observed with other participants exploring CFL.

Fig. 10
figure 10

The heatmaps of the mouse trajectories reveal three navigation strategies followed by the participants

Full size image

7 Discussion

The focus of this work was on designing and building visualizations tailored to support architects and engineers in their daily tasks. In the following, we discuss some of the limitations and lessons learned.

7.1 Limitations

Many design choices made in this project were influenced by the original design of the Opossum Explorer [48, 49], such as the use of star coordinates and barycentric interpolation. This decision stems from the familiarity of domain experts with the Opossum Explorer, as it is already integrated into their daily tasks. The decision to opt for a glyph-based design was driven by the insights from Wortmann et al. [50], who interviewed 186 AEC practitioners and found that 75% typically work with optimization problems featuring ten parameters or fewer. This is consistent with similar findings in the visualization community [16]. However, the star-glyph design has limited scalability when it comes to effectively visualizing more than a dozen parameters. We tested it on datasets with up to nine parameters, which was sufficient for our domain experts and the type of optimization problems they deal with. However, such an assumption might not generalize to datasets from other domains.

Another limitation of our work is the small sample size (\(N = 12\)) in our user evaluation. However, since we designed the framework for a specific target group of users, it is challenging to scale up the number of participants to the extent that it would lead to statistical significance if we were to conduct an in-depth quantitative evaluation. On the positive side, conducting an in-person evaluation with a small sample size provided unique insights, allowed the observation of strategies, and facilitated detailed discussions with experts, aspects not easily achievable in crowd-sourced evaluations [35].

7.2 Lessons learned

The core contribution of this work is characterizing the domain problem through the definition of three layers: data, projection, and visualization. While prior work touched on specific components within these layers, they were not explicitly framed as dimensions or variables of the design space. We contend that such characterization is essential to consider when designing, building, and evaluating visual analytics tools for exploring fitness landscapes and tools for parameter space exploration in general. However, due to this characterization, many decisions have to be made on each dimension. For example, which surrogate models and sampling methods to use, which projection techniques to apply, and what interpolation methods and glyph designs to choose from. This poses a challenge not only to design and build these tools but, more importantly, to get them used by a demographic group who may not be necessarily knowledgeable about all these aspects.

Designing a visualization tool with many options may seem beneficial at first glance, but it might have the opposite effect if the users lack the necessary knowledge to choose and decide between these options. This could lead to frustration and users becoming more skeptical and less trusting of the results. In that case, it is the role of the visualization experts to make these decisions for them during the design phase, ideally by integrating design alternatives in a seamless way. We observed this issue when some of the study participants had to switch between the different visualization, projection, or interpolation options in our visualization framework. On the flip side, a target user group with high visualization literacy would appreciate a variety of options and would prefer to decide and even adjust the parameters themselves. Therefore, it is essential to study the demographic group, assess their visualization literacy, and use such knowledge to guide the design process.

Another lesson we learned is that technically superior techniques are not necessarily the most useful or easiest to interpret. While UMAP is technically superior to SC, most architects we studied rated UMAP lower than SC across the four questions about understandability, interpretability, faithfulness, and trust, especially when paired with CFL. Half of the participants who were exposed to this combination found it challenging to navigate. Notably, the only participant who showed a preference for UMAP over SC was also the only one who indicated familiarity with t-SNE in the demographics questionnaire. It is noteworthy that none of the participants indicated any familiarity with UMAP, and only five out of twelve had some familiarity with SC.

The third lesson we learned is that it is easier to design and build upon the same tools used by experts rather than proposing an entirely new stack of technology. Architects have a strong community around Rhino/Grasshopper. After all, it is the most common platform for architectural design optimization [50]. Trying to force architects out of their territory by developing tools incompatible with Rhino/Grasshopper, will eventually lead them to abandon the tool to preserve their existing workflow. In the same vein, it is essential to establish a line of continuity between the kind of visualizations or visual stimuli the experts see or use in their daily work and what the new tool brings. For example, while we tried to advocate for the use of abstract data visualizations to gain a high-level understanding of the design space, architects ultimately want to “see” and “inspect” the actual 3D structure. Building tools that only rely on abstract data visualization makes them hard to adopt, especially when the users are not knowledgeable about how to interpret these visualizations.

8 Conclusion and future work

We presented a visual analytics framework for the exploration of fitness landscapes in architectural design. In collaboration with domain experts, we identified multiple needs for understanding and interpreting the results of optimization algorithms. The presented framework provides a basis for the exploration of the design parameter space and support for informed decision-making. Our evaluation showed that the visualization is helpful in understanding and navigating multidimensional spaces resulting from optimization algorithms, and even less experienced users can use it to solve domain-related tasks. Future work could involve exploring alternative surrogate models, sampling strategies, and 2D spatial mappings at the data and projection layers, as well as optimizing the order of the parameters in the star-glyph or exploring other glyph designs for encoding uncertainty and handling multiple objective functions, at the visualization layer. Further, quantitative and qualitative evaluations are needed to validate our findings.

In conclusion, we see this presented framework as an initial step toward an interdisciplinary research to support architectural design through visual analytics. Future research should focus on bridging communities, providing means to explain computational models, and supporting design in architecture, engineering, and construction.