Abstract
In architectural design optimization, fitness landscapes are used to visualize design space parameters in relation to one or more objective functions for which they are being optimized. In our design study with domain experts, we developed a visual analytics framework for exploring and analyzing fitness landscapes spanning data, projection, and visualization layers. Within the data layer, we employ two surrogate models and three sampling strategies to efficiently generate a wide array of landscapes. On the projection layer, we use star coordinates and UMAP as two alternative methods for obtaining a 2D embedding of the design space. Our interactive user interface can visualize fitness landscapes as a continuous density map or a discrete glyph-based map. We investigate the influence of surrogate models and sampling strategies on the resulting fitness landscapes in a parameter study. Additionally, we present findings from a user study (N = 12), revealing how experts’ preferences regarding projection methods and visual representations may be influenced by their level of expertise, characteristics of the techniques, and the specific task at hand. Furthermore, we demonstrate the usability and usefulness of our framework by a case study from the architecture domain, involving one domain expert.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
1 Introduction
The architecture, engineering, and construction (AEC) industry plays a key role in how our world will look in the future. It is facing the challenge that buildings, housing, and infrastructure for more than 2.5 billion people will have to be realized in the next decades. An important part of improving AEC is to advance the way architecture is designed and optimized to reduce energy use, greenhouse gas emissions, and waste. However, despite their great potential, computational optimization methods have had a limited impact on architectural design so far [33, 44, 51], especially when compared with other engineering domains where the use of optimization is common practice [14].
Optimization inherently requires a well-defined problem with clear constraints and objective functions. While such elements might be available at a late stage of the design process, it is rarely the case at early-stage design phases, where design is often described as a wicked problem with ill-defined tasks and incomplete data [1, 10, 44]. At this stage, designers have much freedom but little knowledge about the problem at hand [33]. Further, when exploring optimization spaces designers are not only interested in finding the optimal solution but rather in exploring a wide variety of solutions that obtain an acceptable performance and in understanding how the different design parameters interact with each other and influence the performance. In their interview with 18 practitioners from the AEC sector, Bradner et al. [4] found that “... the computed optimum was often used as the starting point for design exploration, not the end product” and that designers need to “... describe the solution and design spaces.”
These characteristics call for human-in-the-loop-based solutions rather than fully automatic ones, which is an essential prerequisite for visual analytics approaches. In particular, techniques for parameter space analysis and exploration offer a great opportunity not only to find an optimal design for a given set of parameters but also to help designers gain a deeper understanding of the design space.
However, designing and building these solutions comes with challenges. In the visualization community, Sedlmair et al. [38] surveyed 21 tools for visual parameter space analysis. The authors identified the limited sampling capabilities (i.e., data acquisition gap), the challenge of building good surrogate models (i.e., data analysis gap), and the issue of visually navigating the multidimensional spaces (i.e., data cognition gap) as the three main challenges when it comes to building these tools. These challenges are amplified in the AEC industry due to the lack of digitization [1, 27]. Most of the existing parametric toolboxes and optimization solvers [5, 6, 43] are limited when it comes to navigating the multidimensional spaces of the optimization (i.e., data cognition gap) and often restricting when it comes to discovering new solutions (i.e., data analysis and acquisition gaps). An exception are performance maps [48, 49], where the authors address the data cognition gap using star coordinates to obtain a 2D projection of the optimization space. Zorn [52] built on top of that concept by exploring the role of sampling methods and surrogate models and how they influence the resulting landscape. While performance maps provide a continuous overview of the optimization space, they are lacking when it comes to revealing the relationships between the design parameters and the objective function. Additionally, star coordinates are linear projections that do not account for local neighborhoods in the data.
Our work builds on and extends previous work [48, 49, 52] by introducing a visual analytics framework (Fig. 1) to explore fitness landscapes of a single-objective optimization function. The main contributions of our work are as follows: (1) A visual analytics framework for exploring fitness landscapes that span across data, projection, and visualization layers. The framework is embedded in an existing workflow for architectural design. (2) A parameter study to investigate the role of surrogate models, sampling strategies, and number of samples in constructing fitness landscapes. (3) A user study involving 12 domain experts to examine user preferences, navigation strategies, and evaluate understandability and interpretability of the visualization and projection layers within our framework. (4) A case study with an expert and data from the architecture domain.
In contrast to other application areas, the AEC domain remains relatively untapped and has yet to harness the full power of visualization and visual analytics [1, 27]. Our paper is a result of a close collaboration with experts from the architecture domain over the span of nine months. We see our work as a step to bridge the gap between the architecture and the visualization communities, providing insights into how visualization can support computational design and the analysis of complex parameter spaces.
2 Related work
Related work comprises computational optimization in the architectural domain and, in general, visualization, exploration, and analysis of multivariate data.
2.1 Architectural design optimization
The architecture domain is not short of optimization solvers [42, 43, 48] or simulation tools [24, 28]. However, most of these tools, while computationally powerful, have limited visualization capabilities when it comes to exploring the resulting optimization and simulation spaces. More than a decade ago, Erhan et al. [12] introduced ViSA, a visual sensitivity analysis approach for parametric modeling. They reached out across domains and argued in favor of visual analytics and the consideration of human cognitive systems when building such tools. StructureFIT [33] and EvoMass [44, 45] are tools for exploring different design options resulting from evolutionary algorithms. Design explorer [5, 6] and Dream Lens [30] are toolkits that offer different sampling strategies with a special focus on design variables analysis and transformation (i.e., sensitivity analysis). While these tools support design space exploration, they rely on design catalogs for presenting the varying design options and/or overlaying the information directly on the 3D structures. We aim to utilize data visualization to gain a high-level understanding of the design space and the nature of the optimization problem.
There are few prior works that utilized abstract data visualizations for exploring optimization spaces. Asl et al. [3] introduced a visual programming interface combined with parallel coordinates for exploring design alternatives resulting from multi-objective optimization. Fuchkina et al. [15] introduced a framework for exploring parametric modeling with node-link diagrams, parallel coordinates plots, and self-organizing maps. In the context of the fitness landscape, Chen at al. [7] and Stasiuk et al. [41] used k-means clustering to find relationships between the design parameters and the objective function and visualized one representative of each cluster. We distinguish ourselves from this previous work in three ways: (1) We focus on single-objective optimization problems. (2) We aim to obtain a direct overview of the fitness landscape that shows the relationships between the design parameters and the objective function rather than inferring these relationships by a few representative samples. (3) We use star coordinates and dimensionality reduction methods to obtain a 2D spatial mapping of the design space. We position ourselves close to the work of Wortmann et al. [48, 49] and Zorn [52], who employed star coordinates and surrogate models to obtain a continuous overview of the fitness landscape. In comparison, we complement the continuous overview with a glyph-based visualization to reveal the relationships between the design parameters and the objective function. Likewise, we complement the star coordinates with UMAP to preserve local neighborhoods in the data. Additionally, we investigate the influence of the number of samples on the resulting fitness landscapes. Furthermore, we carry out an empirical user evaluation and present a case study, which has not been done by previous work.
2.2 Visualization of multivariate data
Despite recent advances [29], parallel coordinates plots (PCPs) [18, 21] and scatterplot matrices (SPLOMs) [2] remain the two most prominent ways for visualizing multivariate or multidimensional data. SPLOMs show pair-wise relationships between the data variables in scatterplots and are familiar to most users. On the downside, relationships between more than two variables are hard to perceive, and the space consumed by the plot grows quadratically with the number of variables. PCPs represent data items as polylines through equidistant vertical axes. In contrast to SPLOMs, the space consumed by PCPs grows proportionally with the number of variables. While PCPs, in principle, are capable of showing the relationships between the variables, they rely on axes ordering [46] and are prone to visual clutter with an increasing number of data items and therefore need clutter reduction methods [11, 34].
To mitigate these problems and obtain a spatial overview, which is missing in both techniques, radial-based projection techniques such as RadViz [20] and star coordinates [23] depict each data item as a point in a 2D radial layout. While the two approaches project the data in a similar way, RadViz introduces a nonlinear normalization step that hinders outlier detection and makes it challenging to find a useful linear mapping of the resulting plot [36]. Star coordinates, in contrast, preserve linearity in the data and, with proper axes calibration, can recover the attributes of the projected data items [37]. Dimensionality reduction (DR) offers an alternative way to obtain a 2D spatial overview of high-dimensional data. Espadoto et al. [13] quantitatively surveyed 44 of these methods across 8 different traits, such as linearity, preservation of neighborhood, complexity, and inverse-transformation ability. Radial projections and DR methods are known to be challenging when it comes to interpreting the projection results. Several techniques have been proposed to mitigate this problem—by showing projection errors [40], displaying data items in the context of the original dimensions [8], or using glyphs [16] as a unit visualization for representing the points in 2D spaces [22, 25].
We position the visualization aspect of our work close to these techniques. On the one hand, we make use of star coordinates and DR methods to obtain an overview of the optimization space. On the other hand, by incorporating glyphs into this overview, we improve the interpretability of the projection results and depict the relationships between the design parameters and the corresponding objective function, which is a desired functionality when exploring optimization spaces. In our work, we took inspiration from Kindlmann [26] and Hlawatsch et al. [19], who used geometric primitives and radar glyphs to visualize a continuous field. In our case, while we share the continuity aspect of the data, we use star glyphs to reach the same goal.
3 Methodology and requirements
We followed the design study methodology [39] to elicit the requirements, design, build, and evaluate our framework. On an abstract level, the full cycle (from requirement analysis to evaluation) was completed twice. The focus of the first cycle was mainly on the visualization layer, whereas the second cycle addressed the data and projection layers. Our requirement elicitation phase drew from the findings obtained by Bradner et al. [4] and Wortmann et al. [50]. These findings were further refined through a series of semi-structured interviews with three experts in visualization, two experts in architectural design optimization, and one software developer; all are co-authors of this manuscript. Each requirement phase was followed by design, implementation, and evaluation phases. The different phases often overlapped, and the process was highly iterative. In total, the project spanned over nine months.
Architectural design often aims to optimize the design parameters with respect to one or more objective functions (i.e., performance criteria). These performance criteria could be, for example, energy usage, daylight exposure, or material usage. The multidimensional design space combined with the objective function(s), when visualized, is often referred to as a fitness landscape or a performance map [49]. We identified the following requirements for building and designing tools to visualize fitness landscapes.
R1: A diverse landscape Although finding the optimum solution is a fundamental requirement, it can be fulfilled without a visualization tool since the optimum is a direct result of the optimization algorithm. Designers often seek not only the optimum but also a multitude of design alternatives that obtain an adequate performance. A diverse landscape with a wide array of solutions is usually highly valued by designers.
R2: Spatial mapping of the design space While the design space can be visualized using various multivariate data visualization methods such as PCPs or SPLOMs, designers tend to think about high-dimensional space in terms of 2D or 3D spatial layout [47, 49]. A spatial representation enhances designers’ ability to identify regions of interest, clusters of similar solutions, and areas of the design space that may be under or over-sampled.
R3: Understanding the design space Uncovering relationships between the design parameters and the performance is an important step toward understanding the design space. Knowing which parameters have a significant impact on the performance is as important as knowing those that have no impact. The former poses a restriction, whereas the latter gives more design freedom. Similarly, uncovering correlations and potential trade-offs among the parameters themselves leads to a deeper understanding of the design space.
R4: 3D structure Having the ability to view the corresponding 3D structure of a solution is a key feature that the visual analytics framework should provide. This is important since not every well-performing solution is a feasible design. Examining the 3D structure represents the final step for determining a solution’s feasibility.
R5: Interactivity and workflow integration The ability to navigate the design space, compare different solutions, and examine them in a seamless way is highly desirable. Moreover, it is crucial to integrate the visual analytics approach into existing design workflows for better adoption and usability. We provide compatibility with Rhino/Grasshopper as it is the most common platform [50] for computational design and optimization and the one used by our domain experts.
4 Visual analytics framework
We designed a visual analytics framework to fulfill the aforementioned requirements. As displayed in Fig. 2, it comprises a data, a projection, and a visualization layer. The data layer includes surrogate models to provide efficient design solutions and sampling strategies for the multivariate data parameters. The projection layer provides different spatial layouts for the data results and the visualization layer displays the resulting fitness landscapes with continuous and glyph-based representations of the data. The user can interact with all three layers to explore the parameter space.
4.1 Data and parameter study
To fulfill requirement R1, we adopt two surrogate models and three sampling strategies introduced by Zorn [52] to obtain a wider array of landscapes. We first introduce the surrogate models and sampling methods, followed by a brief discussion of our findings from the parameter study.
Surrogate models Surrogate models construct a mathematical approximation of the design space based on the optimization parameters and use such a model to predict unknown sample points. Therefore, they are computationally cheap compared to running physics simulations that might take a few hours or days [51]. In our work, we leverage surrogate models by experimenting with (1) a radial basis function interpolation surrogate model trained using RBFOpt optimizer [9] and (2) a surrogate model resulting from a gradient boosted regression tree (GBRT) model trained on a varying number of samples selected using the Latin hypercube sampling strategy. While (1) optimizes its hyperparameters during the optimization process, we use a Bayesian optimization algorithm to optimize the hyperparameters of (2).
Sampling strategies We investigate three common sampling strategies: (1) Optimization-based sampling, (2) Latin hypercube (LH) sampling, (3) hybrid sampling strategy that combines both of the above. Since expensive physics simulations result in long run times, it is unfeasible to sample more than a couple of hundred samples before exploring the design space. Therefore, we limit the number of samples for evaluating the framework to three sets of 100, 500, and 1000 samples, respectively.
Parameter study To investigate the influence of the surrogate model, the sampling strategy, and the number of samples on the resulting fitness landscapes, we conducted a parameter study based on two datasets. The study showed that RBF landscapes with optimization-based samples, while giving us some degree of global overview (exploration), many points are clustered in areas of high performance (exploitation) (see Fig. 3a). Thus, the resulting landscape does not provide a holistic overview of the design space. On the contrary, sampling a multi-dimensional space in an almost random fashion, as in the case of GRBT landscapes with LH samples, is less likely to obtain many high-performing solutions (see Fig. 3b). RBF, when combined with hybrid samples, leads to a diverse landscape (see Fig. 3c) with solutions ranging from high-performing on the left of the map to low-performing ones on the right. Such landscapes might be valuable for designers. For more details on the parameter study and the complete set of results, we refer to the supplemental materials.
4.2 Projection
Projecting the sample points to a 2D plane allows us to obtain an overview of the fitness landscape, which is essential for exploration (R2). We use star coordinates (SC) and uniform manifold approximation and projection (UMAP) [31] to obtain two alternative projections of the fitness landscape. SC is easy to navigate if one is looking to retrieve a solution with a high value at a single or two adjacent parameters on the coordinate system. These solutions are usually located at the outer boundaries of the map in the direction of the corresponding parameter. However, as we move toward the center of the map, it becomes more ambiguous which parameters contributed the most to a given solution since two solutions of different parameter values could be projected close to each other. UMAP, in contrast, preserves local neighborhoods in the data and, therefore, promises to reveal different clusters of similar solutions. Nevertheless, it comes at the cost of being less intuitive to interpret compared to SC, especially without a layout enrichment strategy [22, 40].
4.3 Visualization
To better understand the design space (R3), we provide two alternative representations for fitness landscapes: continuous and glyph-based. The continuous representation offers a more precise depiction of the optimization space, whereas the glyph-based is better at revealing the relationships between the design parameters and the performance value.
Continuous fitness landscape (CFL) To render a CFL, we estimate the performance values to fill in the missing data between the projected sample points. Interpolating the performance values directly is fast but provides no information beyond the sample points because it does not use the underlying model. Instead, we perform interpolation in the parameter space followed by a query to the surrogate model to get the corresponding performance value. We investigate two different methods: (1) Barycentric interpolation relies on a triangulation of the evaluated and projected points, whereas (2) Shepard interpolation takes all projected points into account, weighting them less with increasing distance. Figure 4 shows an example of the two interpolation methods. With a power parameter of \(p=2\) for the Shepard interpolation, we also take points with higher distances into account.
Glyph-based fitness landscape (GFL) CFLs provide an overview of the optimization space but lack the ability to depict the relationships between the design parameters and performance (R3). Hence, we complement the CFL with a glyph-based fitness landscape to fill this gap. The decision to opt for a glyph-based design was motivated by the low number of attributes featured in architectural design optimization problems [50]. While the glyph design space is huge [16], we decided on star glyph as they are easy to understand and fit with the SC projection. Furthermore, star glyphs are very good at forming recognizable shapes and patterns [25], which, in our case, is crucial to maintain the sense of continuity in the fitness landscape. Figure 5a shows a star glyph representing one data item of ten parameters. The corresponding objective value of each data item is double-encoded by the color of the formed star shape and the background color of the glyph. We use a high opacity value for the background color to distinguish the actual values from the maximum values at each parameter. Utilizing the background color allows us to highlight the small parameter values and to visually sustain the sense of continuity when glyphs are packed in the GFL (see Fig. 5b). To construct the entire GFL, we uniformly discretize the CFL. The discretization rate is a user-defined parameter and implicitly defines the glyph size. At lower discretization rates (i.e., big glyph sizes), more data items are aggregated into a single glyph. Depending on the sampling strategy and the projection technique employed, these data items might vary in the parameter values and/or the corresponding objective value. To account for such variability, we utilize the glyph stroke to encode the standard deviation within the glyph.
Using glyphs allows us to visualize the design parameters, encoded by the glyphs, in relation to a single-objective value, encoded by color. The fitness landscape relies on DR methods to obtain an overview of the optimization space. Interpreting such an overview might not be an easy task due to the projection artifacts. Therefore, using glyphs also serves to improve the interpretability of the projection results. While using glyphs to visualize the projection results is not new [22, 25], the novelty is using them to visualize CFLs and extending the design to show multiple levels of detail.
4.4 Interactive framework
For a framework showing fitness landscapes and 3D structures (R4) integrated into the design workflow (R5) of architects, we implemented a visual analytics approach consisting of multiple views that provide an overview and details for different parameter constellations and their respective 3D representation (Fig. 1). The main view at the center can be changed from a continuous view to a grid view, showing the CFL or GFL, respectively. Both representations, as well as different interpolation methods and color scales, can be changed on demand.
The objective value and the parameter view update live with the mouse movement in the main view, providing parameters and objective values in numbers for the currently hovered glyph or point on the map. Similarly, the render view displays the 3D representation when clicking a sample point. When GFL is selected, the user has the option to adjust the resolution level of the grid.
To support the exploration process, we adapted the idea of a minimap that shows the exploration history, indicating where analysts have already inspected the visualization with the cursor. With this visual aid, it is possible to track the exploration process and focus, for instance, on areas that were not investigated in detail. To provide the full spatial context of the exploration history, it is also possible to show the generated density map superimposed on the main view.
Data drill-down In contrast to CFL, GFL requires more screen space to convey the additional information of parameter sets besides the objective value. Glyphs with thick strokes signify a high standard deviation and is typically an indicator to investigate a glyph in more detail. To explore the individual sub-glyphs at different zoom levels, we included two options: (1) An external view for the portal lens (Fig. 1). (2) An in-place portal lens that is displayed directly at the cursor position (Fig. 6). The portal lens is updated by moving the portal cursor on the grid, and the mouse wheel is used to change the semantic zoom level, which is visually indicated by a partial circle around the portal cursor in both options. The external view has the advantage of showing the glyphs in a larger view without occluding the overview. The in-place variant provides a data drill-down without splitting attention between two views on the screen. We implemented both options and found that users less familiar with performance maps preferred the in-place lens. With this design, it is possible to keep an overview of the fitness landscape while drilling down into specific solutions generated by the model.
5 Case study: optimization of coreless filament wound structures
To showcase our approach with real data in collaboration with a domain expert, we present a case study in the context of coreless filament winding (CFW). For detailed information on the procedure, we refer to the supplemental material.
5.1 Coreless filament wound structures
CFW is a manufacturing process that enables the fabrication of highly differentiated, lightweight, resource-effective, and high-performing building components [17]. In CFW, free-spanning fiber rovings are sequentially placed between anchor points, and the fiber net emerges from the fiber–fiber interactions [32], which is the optimization objective to maximize. The parameter space defines the boundary condition, i.e., the placement of the winding anchors and hyperparameters of the winding sequence. Designers want to explore the CFW component design space qualitatively, considering shapes and boundary conditions’ influence on the resulting fiber net. Depending on the amount of fiber interactions, simulation time varies from seconds to minutes.
5.2 Analysis insights
The expert analyzed the RBF map with optimization-based samples. During the analysis, the expert confirmed their hypotheses about the data but also discovered new insights.
5.2.1 Confirming hypothses
High anchor count correlates with higher objectives The domain expert confirmed such a hypothesis by investigating various solutions on the map. The expert commented that “the map looks a bit similar to what I expected... the solutions with high anchor count [one of the parameters] also have a higher objective.” This finding highlighted the importance of this parameter in the optimization process, indicating its direct impact on the quality of solutions.
The shift parameter influences surface curvature After rendering low and high-performing solutions, the domain expert noted a relationship between the shift parameter and the surface curvature of the structure. Lower shift values led to almost straight spanning components, indicating a flatter structure (Fig. 7c), while higher shift resulted in structures with more curvature due to increased fiber interactions (Fig. 7a, b). The expert added “this is also something we observed in the projects we are working with.”
5.2.2 New insights
Radius1, Radius2, and the Performance While using CFL, the expert noticed that the two parameters, radius1 and radius2, fluctuate significantly in the high-performance area. “What I would be curious to know is if there is a relationship between the two,” the expert added. To answer the question, the expert switched back to GFL and noticed that the two parameters must be of similar value to get a high-performing solution. This was considered one of the main insights they obtained during the analysis.
High-performing solutions often feature triangular frames After investigating the polygon side parameter, the expert was surprised that most high-performing solutions have a low polygon side. With this, the two frames of the structure appear more triangular, as in Fig. 7a or b, than circular, as in Fig. 7c. This was the second insight of the analysis. Understanding this relationship between parameters and the geometry of the solutions provides valuable information for designing structures with specific characteristics tailoring the optimization process to achieve desired outcomes.
5.2.3 Observations
Navigating the map The domain expert had no problems interpreting the SC. They had a very good understanding of where to move in the space, looking for certain parameter configurations. They also noted that the high-performing solutions were clustered on the left side of the map with many samples, while the low-performing ones were on the right side with few samples. They commented: “I guess that makes sense because the RBF tries to maximize.”
Visualization The expert appreciated both the CFL and GFL representations: “Actually, what I like more about the glyphs is that my focus always stays in this area [referring to the canvas area]... here [referring to the CFL] it is clearer to see where the samples are.” The expert frequently used the render view to examine the 3D representation of the structures. Additionally, they used the portal to investigate glyphs with thick strokes, indicating a high standard deviation. The expert also favored rendering the CFL using the Shepard interpolation over the barycentric method, citing that the former provided a more fluid and organized visual representation.
6 User study
We aimed to evaluate our approach with the main research question: “How do domain experts explore fitness landscapes under different projections and visualizations?” We conducted a qualitative experiment to probe the different combinations and identify differences in interpretability, task difficulties, applied strategies, and usability.
6.1 Study design and tasks
There are two variables to investigate in our study: the projection (SC vs. UMAP) and the visualization (CFL vs. GFL). Due to the limited number of domain experts, we chose a mixed study design. The goal here was to explore the design alternatives under qualitative aspects rather than by confirmatory, hypothesis-driven design. To probe the design space with respect to these variables, participants were exposed to either both projection methods and both visualization techniques (G1), both projection methods with only one visualization technique (G2), or both visualization techniques with only one projection method (G3). This way, we could probe overall impressions (G1), and more comparison-based feedback for projections (G2) and the visualizations (G3). The evaluation was based on the following tasks:
T1: Projection mental map Participants are provided with two parameters and are asked to approximate the location of a solution on the map, considering parameters having either low or high values.
T2: Design alternatives Participants have to find two or three design alternatives that achieve relatively high performance but differ in their parameter configurations.
T3: Significant parameters We ask participants to identify parameters that have a significant impact on the performance and identify the impact type (positive or negative).
T4: Parameters correlations Participants have to identify correlations between parameter pairs and the performance. Similar to task T3. They are also asked to identify the type of correlation.
Task T1 was designed to assess participants’ ability to comprehend and navigate the fitness landscape under the different projection methods. Considering its fundamental role in solving subsequent tasks, it was placed at the beginning of the evaluation and repeated twice. We refer to the two trials as T1.1 and T1.2. The selection of tasks T2–T4 was motivated by understanding the design space (R3).
6.2 Procedure
Each task started with a brief introduction and ended with a difficulty rating on a 5-point Likert scale, a description of applied strategies, and preferred projection and/or visualization. In the end, we held an open discussion where we asked participants to rate understandability, interpretability, faithfulness, and trust of the projection method and/or visualization techniques they were exposed to. In total, we recruited \(N = 12\) participants (four identified as females) equally divided into three groups (i.e., four per group). All had a university degree in architecture (six master’s, five bachelor’s, and one Ph.D.) and eight indicated some familiarity with the Performance Map technique. The study received approval from the ethics committee of the university and participants were compensated at the rate of EUR 12 per hour. For more details regarding the study procedure, questionnaires, dataset, and stimuli, we refer to the supplemental material. In the following, we discuss the results.
6.3 Tasks difficulty and preferences
Figure 8 shows the subjective rating of participants regarding the difficulty of each task. Task T3 stands out as the most difficult task. T4 was rated less difficult; this could be due to some learning effects. Tasks T1 and T2 were mostly rated easy to solve. Figure 9 shows the projection and visualization preferences of the participants. For the projection, we only considered the data from the study groups G1 and G2 since those participants had a choice to switch between the two projection methods. Likewise, for the visualization, we only considered the data from G1 and G3.
SC versus UMAP SC was the preferred projection method across all tasks. Participants used the words “easy to understand,” “easy to locate solutions,” “quick to learn,” and “straightforward” when asked to explain their preference. They also often referred to the axes of SC as the key that made their job a bit easier. UMAP did not have such axes in comparison, and therefore, it was harder to navigate the landscape. One participant stated: “I felt that CFL-UMAP showed more solutions, but I had to go hunting for them because I did not know where to start my search.” Participants favoring UMAP over SC mentioned reasons that were not necessarily related to how they read or understood the projection. Instead, they sometimes confused the influence of the projection method with the influence of the visualization method (i.e., when changing from GFL-SC to CFL-UMAP).
One participant, when asked about why they favored UAMP in task T1 answered: “In this combination [referring to CFL-UMAP] I had quick success.” They elaborated that the choice of projection method did not impact their decision: “I was ignoring what is happening there [referring to the canvas]... that was only a loose connection.” One participant favored UMAP in task T2 as it showed them more clusters than SC. They said: “SC was more confusing as only one cluster was visible. In UMAP, I can see three or four clusters.” The option “not sure, both” was frequently chosen by participants after attempting to solve tasks T3 and T4 (see Fig. 9a). Participants often switched between the projection methods to try to find an answer. One participant, however, chose this option in task T1 since SC showed them a solution in a location that did not align with their understanding of how SC works. This participant stated that “axis was confusing, even though I knew what it meant ... would rather not have an axis.” Fig. 9c shows the preferred projection grouped by visualization. On most occasions (13/15), when participants were exposed exclusively to CFL (group G2) or showed a preference for CFL over GFL when exposed to both (group G1), they also tended to favor SC over UMAP.
CFL versus GFL Figure 9b shows the participants’ preferences regarding the visualization technique. GFL and CFL were rated equally in tasks T1 and T2. In T3, while CFL was favored over GFL, participants were undecided. This, similar to the projection results (Fig. 9a), reflects the difficulty of T3, where participants had to switch between the two representations. T4, in contrast, clearly indicates a preference for GFL. When participants preferred CFL over GFL, they often pointed out that CFL was better in terms of distinguishing “color gradient” or “color brightness” compared to GFL. In contrast, participants who favored GFL often highlighted their ability to distinguish and compare “shapes” or “geometry” of different solutions, especially in T3 and T4.
Several participants came to the conclusion that both representations have their strengths. While CFL is better suited for exploratory analysis, GFL performs better regarding detailed investigation. One participant commented: “CFL more exploratory but GFL is more exact.” Another participant added: “to play around to find a spot, the first one [CFL] is easier to use. To have more information, it seems like the other one is more beneficial.” This participant also drew an analogy, that looking for a high-performing solution in CFL is like playing a game of blindfolded candy hunting. They added: “I just need to find one... but with a huge lack of certainty.” Fig. 9d shows the visualization preferences grouped by projection method. While the results suggest a tendency toward GFL when paired with UMAP, further investigation is needed to conclusively confirm such a hypothesis. Participants favored GFL over CFL, especially with barycentric interpolation. It produces triangular artifacts (Fig. 4), which made it less appealing when paired with CFL. Some participants expressed that these artifacts made them trust the visualization less.
6.4 Navigation strategies
Participants explored the maps by moving the mouse cursor on the main canvas and observing the changes in the other views. During the study, we observed three navigation patterns. These patterns were later confirmed by analyzing the heatmaps of the participants’ mouse trajectories. For more details, see the supplemental materials.
Moving a long axes Some of the participants who were exposed to SC projection started the exploration by moving in a circular motion around the SC axes. Once they found something interesting, they stopped and started moving up and down along the axis of interest. This strategy was often used when trying to solve tasks T3 and T4. This also can be seen in the heatmap of the mouse trajectories of one of the participants as in Fig. 10a.
Random search Randomly exploring the map was often done by participants when UMAP was used as the underlying projection. Nevertheless, other participants followed this strategy with SC when their reasoning about SC failed to deliver an answer (see Fig. 10b).
Systematic search Few participants preferred to scan column-by-column or row-by-row in a systematic manner (see Fig. 10c). While this behavior was more pronounced with GFL due to the grid layout, it was also observed with other participants exploring CFL.
7 Discussion
The focus of this work was on designing and building visualizations tailored to support architects and engineers in their daily tasks. In the following, we discuss some of the limitations and lessons learned.
7.1 Limitations
Many design choices made in this project were influenced by the original design of the Opossum Explorer [48, 49], such as the use of star coordinates and barycentric interpolation. This decision stems from the familiarity of domain experts with the Opossum Explorer, as it is already integrated into their daily tasks. The decision to opt for a glyph-based design was driven by the insights from Wortmann et al. [50], who interviewed 186 AEC practitioners and found that 75% typically work with optimization problems featuring ten parameters or fewer. This is consistent with similar findings in the visualization community [16]. However, the star-glyph design has limited scalability when it comes to effectively visualizing more than a dozen parameters. We tested it on datasets with up to nine parameters, which was sufficient for our domain experts and the type of optimization problems they deal with. However, such an assumption might not generalize to datasets from other domains.
Another limitation of our work is the small sample size (\(N = 12\)) in our user evaluation. However, since we designed the framework for a specific target group of users, it is challenging to scale up the number of participants to the extent that it would lead to statistical significance if we were to conduct an in-depth quantitative evaluation. On the positive side, conducting an in-person evaluation with a small sample size provided unique insights, allowed the observation of strategies, and facilitated detailed discussions with experts, aspects not easily achievable in crowd-sourced evaluations [35].
7.2 Lessons learned
The core contribution of this work is characterizing the domain problem through the definition of three layers: data, projection, and visualization. While prior work touched on specific components within these layers, they were not explicitly framed as dimensions or variables of the design space. We contend that such characterization is essential to consider when designing, building, and evaluating visual analytics tools for exploring fitness landscapes and tools for parameter space exploration in general. However, due to this characterization, many decisions have to be made on each dimension. For example, which surrogate models and sampling methods to use, which projection techniques to apply, and what interpolation methods and glyph designs to choose from. This poses a challenge not only to design and build these tools but, more importantly, to get them used by a demographic group who may not be necessarily knowledgeable about all these aspects.
Designing a visualization tool with many options may seem beneficial at first glance, but it might have the opposite effect if the users lack the necessary knowledge to choose and decide between these options. This could lead to frustration and users becoming more skeptical and less trusting of the results. In that case, it is the role of the visualization experts to make these decisions for them during the design phase, ideally by integrating design alternatives in a seamless way. We observed this issue when some of the study participants had to switch between the different visualization, projection, or interpolation options in our visualization framework. On the flip side, a target user group with high visualization literacy would appreciate a variety of options and would prefer to decide and even adjust the parameters themselves. Therefore, it is essential to study the demographic group, assess their visualization literacy, and use such knowledge to guide the design process.
Another lesson we learned is that technically superior techniques are not necessarily the most useful or easiest to interpret. While UMAP is technically superior to SC, most architects we studied rated UMAP lower than SC across the four questions about understandability, interpretability, faithfulness, and trust, especially when paired with CFL. Half of the participants who were exposed to this combination found it challenging to navigate. Notably, the only participant who showed a preference for UMAP over SC was also the only one who indicated familiarity with t-SNE in the demographics questionnaire. It is noteworthy that none of the participants indicated any familiarity with UMAP, and only five out of twelve had some familiarity with SC.
The third lesson we learned is that it is easier to design and build upon the same tools used by experts rather than proposing an entirely new stack of technology. Architects have a strong community around Rhino/Grasshopper. After all, it is the most common platform for architectural design optimization [50]. Trying to force architects out of their territory by developing tools incompatible with Rhino/Grasshopper, will eventually lead them to abandon the tool to preserve their existing workflow. In the same vein, it is essential to establish a line of continuity between the kind of visualizations or visual stimuli the experts see or use in their daily work and what the new tool brings. For example, while we tried to advocate for the use of abstract data visualizations to gain a high-level understanding of the design space, architects ultimately want to “see” and “inspect” the actual 3D structure. Building tools that only rely on abstract data visualization makes them hard to adopt, especially when the users are not knowledgeable about how to interpret these visualizations.
8 Conclusion and future work
We presented a visual analytics framework for the exploration of fitness landscapes in architectural design. In collaboration with domain experts, we identified multiple needs for understanding and interpreting the results of optimization algorithms. The presented framework provides a basis for the exploration of the design parameter space and support for informed decision-making. Our evaluation showed that the visualization is helpful in understanding and navigating multidimensional spaces resulting from optimization algorithms, and even less experienced users can use it to solve domain-related tasks. Future work could involve exploring alternative surrogate models, sampling strategies, and 2D spatial mappings at the data and projection layers, as well as optimizing the order of the parameters in the star-glyph or exploring other glyph designs for encoding uncertainty and handling multiple objective functions, at the visualization layer. Further, quantitative and qualitative evaluations are needed to validate our findings.
In conclusion, we see this presented framework as an initial step toward an interdisciplinary research to support architectural design through visual analytics. Future research should focus on bridging communities, providing means to explain computational models, and supporting design in architecture, engineering, and construction.
Data availibility
Research data and the supplemental materials can be downloaded from the following https://doi.org/10.18419/darus-4164.
References
Abdelaal, M., Amtsberg, F., Becher, M., Estrada, R.D., Kannenberg, F., Calepso, A.S., Wagner, H.J., Reina, G., Sedlmair, M., Menges, A., Weiskopf, D.: Visualization for architecture, engineering, and construction: shaping the future of our built world. IEEE Comput. Graphics Appl. 42(2), 10–20 (2022)
Andrews, D.F.: Plots of high-dimensional data. Biometrics pp. 125–136 (1972)
Asl, M.R., Bergin, M., Menter, A., Yan, W.: BIM-based parametric building energy performance multi-objective optimization. In: Proceedings of the 32nd eCAADe Conference, pp. 455–464 (2014)
Bradner, E., Iorio, F., Davis, M., et al.: Parameters tell the design story: ideation and abstraction in design optimization. In: Proceedings of the Symposium on Simulation for Architecture & Urban Design, vol. 26 (2014)
Brown, N.C., Jusiega, V., Mueller, C.T.: Implementing data-driven parametric building design with a flexible toolbox. Autom. Construct. 118, 103252 (2020)
Brown, N.C., Mueller, C.T.: Design variable analysis and generation for performance-based parametric modeling in architecture. Int. J. Archit. Comput. 17(1), 36–52 (2019)
Chen, K.W., Janssen, P., Schlueter, A.: Analysing populations of design variants using clustering and archetypal analysis. In: Proceedings of the 33rd eCAADe Conference, pp. 251–260 (2015)
Cheng, S., Mueller, K.: The data context map: fusing data and attributes into a unified display. IEEE Trans. Visual Comput. Graph. 22(1), 121–130 (2015)
Costa, A., Nannicini, G.: RBFOpt: an open-source library for black-box optimization with costly function evaluations. Math. Program. Comput. 10(4), 597–629 (2018)
Cross, N.: Designerly ways of knowing. Des. Stud. 3(4), 221–227 (1982)
Ellis, G.P., Dix, A.J.: Enabling automatic clutter reduction in parallel coordinate plots. IEEE Trans. Visual Comput. Graph. 12(5), 717–724 (2006)
Erhan, H., Salmasi, N.H., Woodbury, R.: Visa: a parametric design modeling method to enhance visual sensitivity control and analysis. Int. J. Archit. Comput. 8(4), 461–483 (2010)
Espadoto, M., Martins, R.M., Kerren, A., Hirata, N.S., Telea, A.C.: Toward a quantitative survey of dimension reduction techniques. IEEE Trans. Visual Comput. Graph. 27(3), 2153–2173 (2019)
Flager, F., Haymaker, J.: A comparison of multidisciplinary design, analysis and optimization processes in the building construction and aerospace industries. In: 24th International Conference on Information Technology in Construction, pp. 625–630. Maribor Slovenia (2007)
Fuchkina, E., Schneider, S., Bertel, S., Osintseva, I.: Design space exploration framework. eCAADe 36, 367–376 (2018)
Fuchs, J., Isenberg, P., Bezerianos, A., Keim, D.: A systematic review of experimental studies on data glyphs. IEEE Trans. Vis. Comput. Graph. 23(7), 1863–1879 (2017)
Gil Pérez, M., Zechmeister, C., Kannenberg, F., Mindermann, P., Balangé, L., Guo, Y., Hügle, S., Gienger, A., Forster, D., Bischoff, M., Tarín, C., Middendorf, P., Schwieger, V., Gresser, G., Menges, A., Knippers, J.: Computational co-design framework for coreless wound fibre-polymer composite structures. J. Comput. Des. Eng. 9(2), 310–29 (2022)
Heinrich, J., Weiskopf, D.: State of the art of parallel coordinates. In: M. Sbert, L. Szirmay-Kalos (eds.) Eurographics 2013—state of the art reports. The Eurographics Association (2013)
Hlawatsch, M., Leube, P., Nowak, W., Weiskopf, D.: Flow radar glyphs-static visualization of unsteady flow with uncertainty. IEEE Trans. Visual Comput. Graphics 17(12), 1949–1958 (2011)
Hoffman, P., Grinstein, G., Marx, K., Grosse, I., Stanley, E.: DNA visual and analytic data mining. In: Proceedings of Visualization’97, pp. 437–441. IEEE (1997)
Inselberg, A.: The plane with parallel coordinates. Vis. Comput. 1, 69–91 (1985)
Kammer, D., Keck, M., Gründer, T., Maasch, A., Thom, T., Kleinsteuber, M., Groh, R.: Glyphboard: visual exploration of high-dimensional data combining glyphs with dimensionality reduction. IEEE Trans. Visual Comput. Graph. 26(4), 1661–1671 (2020)
Kandogan, E.: Star coordinates: a multi-dimensional visualization technique with uniform treatment of dimensions. In: Proceedings of the IEEE Information Visualization Symposium, vol. 650, p. 22 (2000)
Karamba3D. https://karamba3d.com/. Accessed on 26 Mar 2023
Keck, M., Kammer, D., Gründer, T., Thom, T., Kleinsteuber, M., Maasch, A., Groh, R.: Towards glyph-based visualizations for big data clustering. In: Proceedings of the 10th International Symposium on Visual Information Communication and Interaction, pp. 129–136 (2017)
Kindlmann, G.: Superquadric tensor glyphs. In: Proceedings of the Sixth Joint Eurographics-IEEE TCVG Conference on Visualization, pp. 147–154 (2004)
Knippers, J., Kropp, C., Menges, A., Sawodny, O., Weiskopf, D.: Integrative computational design and construction: rethinking architecture digitally. Civ. Eng. Des. 3(4), 123–135 (2021)
Ladybug tools: home page. https://www.ladybug.tools/. (Accessed on 03/26/2023)
Liu, S., Maljovec, D., Wang, B., Bremer, P.T., Pascucci, V.: Visualizing high-dimensional data: advances in the past decade. IEEE Trans. Visual Comput. Graph. 23(3), 1249–1268 (2016)
Matejka, J., Glueck, M., Bradner, E., Hashemi, A., Grossman, T., Fitzmaurice, G.: Dream lens: exploration and visualization of large-scale generative design datasets. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI ’18, pp. 1–12. Association for Computing Machinery (2018)
McInnes, L., Healy, J., Melville, J.: UMAP: uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426 (2018)
Menges, A., Kannenberg, F., Zechmeister, C.: Computational co-design of fibrous architecture. Archit. Intell. 1(1), 6 (2022)
Mueller, C., Ochsendorf, J.: From analysis to design: A new computational strategy for structural creativity. In: Proceedings of the 2nd International Workshop on Design in Civil and Environmental Engineering, pp. 46–56. Mary Kathryn Thompson (2013)
Rodrigues, N., Schulz, C., Lhuillier, A., Weiskopf, D.: Cluster-flow parallel coordinates: tracing clusters across subspaces. In: Graphics Interface 2020 (2020)
Ross, J., Irani, L., Silberman, M.S., Zaldivar, A., Tomlinson, B.: Who are the crowdworkers? Shifting demographics in Mechanical Turk. In: CHI ’10 Extended Abstracts on Human Factors in Computing Systems, pp. 2863–2872 (2010)
Rubio-Sánchez, M., Raya, L., Diaz, F., Sanchez, A.: A comparative study between Radviz and star coordinates. IEEE Trans. Visual Comput. Graph. 22(1), 619–628 (2015)
Rubio-Sánchez, M., Sanchez, A.: Axis calibration for improving data attribute estimation in star coordinates plots. IEEE Trans. Visual Comput. Graph. 20(12), 2013–2022 (2014)
Sedlmair, M., Heinzl, C., Bruckner, S., Piringer, H., Möller, T.: Visual parameter space analysis: a conceptual framework. IEEE Trans. Visual Comput. Graph. 20(12), 2161–2170 (2014)
Sedlmair, M., Meyer, M., Munzner, T.: Design study methodology: reflections from the trenches and the stacks. IEEE Trans. Visual Comput. Graph. 18(12), 2431–2440 (2012)
Stahnke, J., Dörk, M., Müller, B., Thom, A.: Probing projections: interaction techniques for interpreting arrangements and errors of dimensionality reductions. IEEE Trans. Visual Comput. Graph. 22(1), 629–638 (2015)
Stasiuk, D., Thomsen, M.R., Thompson, E.: Learning to be a vault: implementing learning strategies for design exploration in inter-scalar systems. In: Proceedings of the 32nd eCAADe Conference, pp. 381–390 (2014)
Vierlinger, R.: Multi objective design interface. Master’s thesis, University of Applied Arts Vienna (2013)
Wallacei: Evolutionary engine for Grasshopper 3D. https://www.wallacei.com/. Accessed on 03 June 2023
Wang, L.: Workflow for applying optimization-based design exploration to early-stage architectural design-case study based on EvoMass. Int. J. Archit. Comput. 20(1), 41–60 (2022)
Wang, L., Chen, K., Janssen, P., Ji, G.: Enabling optimisation-based exploration for building massing design: a coding-free evolutionary building massing design toolkit in Rhino-Grasshopper. In: RE: Anthropocene, Design in the Age of Humans-Proceedings of the 25th International Conference on Computer-Aided Architectural Design Research in Asia (CAADRIA 2020), vol. 1, pp. 255–264 (2020)
Wegman, E.J.: Hyperdimensional data analysis using parallel coordinates. J. Am. Stat. Assoc. 85(411), 664–675 (1990)
Wise, J.A., Thomas, J.J., Pennock, K., Lantrip, D., Pottier, M., Schur, A., Crow, V.: Visualizing the non-visual: spatial analysis and interaction with information from text documents. In: Proceedings of Visualization 1995 Conference, pp. 51–58. IEEE (1995)
Wortmann, T.: Opossum-introducing and evaluating a model-based optimization tool for Grasshopper. In: Proceedings of the CAADRIA, vol. 17, pp. 283–292 (2017)
Wortmann, T.: Surveying design spaces with performance maps: a multivariate visualization method for parametric design and architectural design optimization. Int. J. Archit. Comput. 15(1), 38–53 (2017)
Wortmann, T., Cichocka, J., Waibel, C.: Simulation-based optimization in architecture and building engineering-results from an international user survey in practice and research. Energy Build. 259, 111863 (2022)
Wortmann, T., Costa, A., Nannicini, G., Schroepfer, T.: Advantages of surrogate models for architectural design optimization. Artif. Intell. Eng. Des. Anal. Manuf. 29(4), 471–481 (2015)
Zorn, M.B.: A novel software framework for architectural design space exploration. In: Proceedings of the 34th Forum Bauinformatik, p. 357–364. Ruhr-Universität Bochum, Universitätsbibliothek (2023)
Funding
Open Access funding enabled and organized by Projekt DEAL. This work was supported by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy – EXC 2120/1 – 390831618. Open Access funding enabled and organized by Projekt DEAL.
Author information
Authors and Affiliations
Contributions
Conceptualization: MA, MZ, TW, DW, and KK; Data Curation: MA, MG, MZ, and FK; Formal Analysis: MA, MG, MZ, and FK; Funding Acquisition: AM, TW, DW, and KK; Investigation: MA, MG, MZ, and FK; Methodology: MA, MZ, DW, and KK. Project Adminstration: MA, DW, and KK; Resources: MZ, FK, AM, and TW; Software: MA, MG, and MZ; Supervision: MA, MZ, AM, TW, DW, and KK; Validation: MA and DW; Visualization: MA, MG, FK, and KK; Writing - original draft: MA, MG, MZ, FK, TW, DW, and KK; Writing - review & editing: MA, DW, and KK.
Corresponding author
Ethics declarations
Conflict of interest
The authors declare no competing interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Abdelaal, M., Galuschka, M., Zorn, M. et al. Visual analysis of fitness landscapes in architectural design optimization. Vis Comput 40, 4927–4940 (2024). https://doi.org/10.1007/s00371-024-03491-3
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00371-024-03491-3