Using large-scale augmented floor surfaces for industrial applications and evaluation on perceived sizes

Abstract

Large high-resolution displays (LHRDs) provide an enabling technology to achieve immersive, isometrically registered, virtual environments. It has been shown that LHRDs allow better size judgments, higher collaboration performance, and shorter task completion times. This paper presents novel insights into human size perception using large-scale floor displays, in particular in-depth evaluations of size judgment accuracy, precision, and task completion time. These investigations have been performed in the context of six, novel applications in the domain of automotive production planning. In our studies, we used a 54-sqm sized LED floor and a standard tablet visualizing relatively scaled and true to scale 2D content, which users had to estimate using different aids. The study involved 22 participants and three different conditions. Results indicate that true to scale floor visualizations reduce the mean absolute percentage error of spatial estimations. In all three conditions, we did not find the typical overestimation or underestimation of size judgments.

Introduction

Large high-resolution displays (LHRDs) provide an enabling technology to achieve immersive, isometrically registered, virtual environments. Wall-sized visualization hardware is getting more and more common in research, entertainment, public signage—and of course industry. The reasons are constantly increasing technical specifications while prices drop at the same time. This trend especially applies for the two most common ways of wall-sized visualization hardware: LED-walls and projector systems. Both LHRD technologies are able to display large amounts of information, for example alphanumeric data, 3D CAD files, or immersive experiences. LHRDs can also be found at increasing numbers of public signage installations as shared interaction spaces. Their content not only to be consumed by their visitors but also presenting new ways of interaction and collaboration (see Peltonen et al. [34]). Even more, immersive display technologies are rumored to allow more accurate size judgments, better collaboration performance in workshops, and less task completion time [14]. However, according to Andrews et al. [4], the advantages of additional pixels—generated by high pixel densities and large installations—can only be realized “through understanding of the interaction between visualization design, perception, interaction techniques, and the display technology” [4]. Therefore, this publication focuses on these aspects of large-scale augmented floor surfaces:

• Application scenarios for automotive production planning: An in-depth literature review on application scenarios for LHRDs and augmented floor surfaces shows that industrial applications have yet not been researched thoroughly. Six application scenarios are presented in this publication and discussed.

• Evaluation on perceived sizes: As the literature review shows that human size perception using augmented floor surfaces has not been in research focus extensively, this publication presents a study on this topic. Bezerianos et al. presented a call for research[8] on the perception of data on wall-sized displays, since there is still little research carried out in this domain: “We do not yet know how the perceptual affordances of a wall, such as the wide viewing angles they cover, affect how data is perceived and comprehended.” We follow this call for research and focus on 2D data visualization on large-scale LED floor displays showing contents in absolute scale.

State of the art in LHRD visualization systems

Large high-resolution displays (LHRDs) are used to display large amounts of data either for alphanumeric, graphics (2D) or 3D data. In 1991, Mark Weiser published an article on “The Computer for the 21st century” [47] and proposed three different clusters of visualization devices. He clustered these devices in three different groups namely “tab,” “pad,” and “board”-sized devices. These “boards” are defined to be yard-sized (> 91 cm) displays and he proposes them to be used “in the home, video screens, and bulletin boards; in the office, bulletin boards, white boards, or flip charts.” In 2009, Terrenghi et al. extend Weiser’s size taxonomy of displays to “Inch,” “Foot,” Yard,” “Perch,” and “Chain,” since all of them differ in their form of social interaction [43] within the multi-person-display ecosystems. Nevertheless, still in 2015, Lischke et al. [26] come to the conclusion that digital “boards” are still rarely used. However, there would be a good chance that wall-sized display-“boards” will become commonplace within the next decade, like smartphones and tables did in the last decade” [26].

Industrial production validation scenarios in automotive industry require interactive collaborative spaces and large display devices. Such combinations of multiple visualization devices are called multi-display environments (MDEs). Garcia-Sanjuan et al. present a general taxonomy of MDEs, classifying their topology with respect to “homogeneity of surfaces,” “spatial form,” “regularity of shape,” “size,” “mobility,” and “scalability” [20] and also present many use cases in this literature review. Following Lischke et al. [26], software is the key enabler for simple setup of multi-display environments and easy usability of these systems, which is still a hindrance factor for broad use. As parallelization of workflows get more and more important, LHRDs allow for visualizing complex data or switching between tasks without hiding required information at the same time. Rogers and Lindley present a study on collaboration at vertical and horizontal large displays [38]. They found that the physical arrangement of publicly shared displays has influence on the social roles between collaboration members, such as switching roles more frequently, greater awareness for each other, and exploration of more ideas. These findings by Rogers and Lindley, Lischke et al., and Garcia-Sanjuan et al. have directly influenced the automotive production planning application scenarios, presented in the latter.

Additionally, industrial use cases aim for a high efficiency when using LHRDs. In 2003, Czerwinski et al. published a study on the performance using large-scale displays in comparison with regular sized desktop screens [14]. They discovered that the user’s task completion time and productivity can be significantly increased for specific tasks by using larger visualization techniques. Analogously, in 2009, Bi and Balakrishnan supported these findings by using even larger LHRDs [9]. In their week-long survey, they supervised users working with a LHRD and discovered that LHRDs enhance the user’s awareness for peripheral applications, facilitate multi-window and rich information tasks, and provide an immersive experience. “The results indicate that users unanimously prefer using a large display” [9]. Interestingly, Bi and Balakrishnan observed that “users tend to utilize the center part as the focal region and the remaining space as the peripheral region. The results also reveal that users on a large displays perform more window moving and resizing, but less minimizing and maximizing operations as compared with a single- or dual-monitor” [9].

Content representation with LHRDs

Having physically installed an LHRD system, users want to make best use of its capabilities. Therefore, adapted content representation and interaction for this type of visualization devices is crucial. Andrews et al. present design considerations, outline challenges, and future opportunities for designing visualizations on LHRDs [4]. They analyze physical display technologies, visual encoding, visualization designs, and user interaction. Andrews et al. describe the benefit of additional pixels with the following, explicitly non-exhaustive, list:

• More data entities

• Ability to show greater data dimensionality

• More data details

• Multi-scale data

• More data complexity or heterogeneity

• Space for processes

• Space for sense-making

• Enable collaboration with private and shared spaces

Complex data and large amounts of data are visualized on LHRDs. Lischke et al. present a study and show that using such systems enable humans to scan large areas quickly for objects and visual cues [28]. For windowed applications, Lischke et al. explore the design space of LHRDs in 2017 by proposing four different graphical interfaces to be displayed in such arrangements [27]. They focus on windowed arrangements and admit that it is still a challenging task. They propose four new alignment techniques, namely “curved zooming, window grouping, window spinning, and side pane navigation” and summarize their work with exploring the design space for focus switching but keeping spatial relations for related windows contents.

Andrews et al. examine how LHRDs support sense making and how the increased space affects the cognitively demanding task of sense making [3]. They explore and show that such a spatial environment supports “sense making by becoming part of the distributed cognitive process” and found “clear evidence of analysts using the space both as a form of rapid access external memory and as an added semantic layer providing both external memory and a semantic layer.” This flexible semantic layer adds meaning to the displayed information such as ordering, proximity, and alignment for clusters. This leads to a reduced need “for elaborate internal models by replacing memorization and computation with perception.”

Observing the content representation and interaction with the graphical interfaces on LHRDs, Lischke et al. conclude that completely novel concepts are required to present content on LHRDs. The classic ways of arranging and hiding windows are neither suitable nor required anymore, since visualization can be parallelized thanks to the way larger resolutions.

Interaction with LHRDs

Lischke et al. also argue that “the success of wall-sized display installations highly depends on the interaction technique used in the particular setup” [26] and propose to get “a clear understanding of advantages and disadvantages of interaction techniques” used with LHRDs. All these findings are required in the industrial sector, as validation scenarios typically rely on commercially available software, which are not natively built for LHRDs. Industrial applications want to leverage the benefits of showing multi-scale data entities and massive amounts of data in parallel, such as product, process, and resource information at the same time.

For LHRDs, multiple input devices can be utilized: Classical interfaces (e.g., mouse and keyboard), natural user interfaces like direct or indirect touch (such as pointing or clicking by Vogel et al. [45] and Malik et al. [30]), 3D interaction devices (e.g., Microsoft Kinect DK), and multi-device strategies (body-attached interaction devices—smartphones [39], glasses, etc.).

LHRD augmented floor surfaces

In contrast to common LHRD wall setups, LHRD floor systems and their applications have not been vastly in scope of research yet. LHRD floors are also called “augmented floor surfaces” and “floor visualization systems” in literature. It is the only form factor, where an infinite scaleability for visualization and direct touch can be achieved due to its horizontal alignment.

Two groundbreaking works have to be mentioned in the context of augmented floor surfaces. In 1993, Cruz-Neira et al. presented the “Cave Automated Virtual Environment” (CAVE) also including a floor display for immersive environments. Cruz-Neira et al. used their “floor wall” projection system for the first presentation of a CAVE setup for virtual reality (VR) applications [13]. Another important work has been presented by Pinhanez in 2001. This novel projection system utilizes a rotating mirror to augment all areas of a room, including the floor surfaces [36]. These setups inspired many further research activities based on these works.

In literature, augmented floor surfaces are set up in various ways. The following, non-exhaustive list clusters these form factors:

• Back projection systems e.g., MultiToe by Augsten et al. [6]

• Front projection systems e.g., laser projection-based system by Müller et al. [31] and Everywhere Display projection by Pinhanez [36]

• LED-based systems e.g., by Dalton et al. [15]

• Low-poly lumination systems e.g., hexagon arrangement by Delbrück et al. [16]

• Irregular lumination systems Camba et al. [11]

• Peripheral halos Vermeulen et al. [44]

Application scenarios for augmented floor surfaces

Research has presented only a few application scenarios for augmented floor surfaces yet, most in the domain of entertainment and gaming such as ShareVR [22], MultiToe [6], IGameFloor [21], SpaceHopper [19], or Kickables [41]. For outdoor advertisement in public spaces, Camba et al. presented a [11] tiled floor visualization system with irregular lumination realized with optical fiber rods. This makes the low resolution of 6×6 LEDs look more interesting. Additionally, they give a general overview on tactile floor setups. In the domain of health and sports use cases, Heller et al. [23] present a smartfloor for motivating people to do more sports by using a floor projection with an interactive floor cells. Interaction with the games on the floor is carried out with 50 cm × 50 cm force weight cells. They present 3 different games “tightrope,” “smartdance,” and “pong” each for encouraging people to work out.

Petersen et al. [35] present design considerations for floor interaction in architectural environments. They present three different interactive floor concepts and use them to derive design issues for interactive floors. They divide the design space into “plaza interaction” and “street interaction.” For plaza interaction, no dominant direction is given for the content, due to the multi-directional access vectors. Street interaction on the other hand is defined by unidirectional access. Hence, more efficient interaction can be assumed.

Law et al. present a multi-modal floor for immersive environments [25] in 2009. They combine auditory, tactile, and visual feedback of the users’ steps in order to create the impression of “walking over natural ground surfaces, such as snow and ice.” The authors argue, by just presenting visual and auditory feedback and leaving out tactile feedback, creates a perceptual conflict, which lacks the desired immersion.

Vermeulen et al. are giving insights into dynamic peripheral floor visualizations with isometrically registered tracking systems [44]. They explore the design space and discuss design considerations for peripheral floor visualizations to convey the users’ information on the tracking fidelity of a system, to show up borders and interaction zones, and to give cues to invite users for spatial movements. These kind of design considerations, especially when connecting and mediating interactions with primary interaction devices.

Schmidt et al. argue that the price of having a large-scale floor visualization with direct touch induces bad ergonomics for users [40]. They evaluate ergonomics while interacting with a floor visualization system and derive a novel system to interactively adapt the content to the operator’s pose in which the user interacts with the floor. This pose-aware system enables a smooth transition between views and is a countermeasure for the “prolonged standing, especially in combination with looking down, quickly causes fatigue and repetitive strain” [40]. The interaction design even gives multiple users the possibility to adapt their personal view to their optimal individual body posture.

Interaction focused research is presented by Schmidt et al., as they present a set of foot-based interaction tangibles, called “Kickables” [41]. They are intended to be used for “very large interaction surfaces.” A set of tangibles for certain UI controls are proposed and evaluated with the affordances they take, such as knobs, switches, sliders, and radio buttons. They propose them to be used for walk up installations.

The literature review reveals that by now, there are no research publications on industrial use cases utilizing large augmented floor visualizations, besides the author’s own previous publication on true scale visualizations for automotive production planning [33] using a scalable floor projection system. As there is a huge potential of using these augmented floor surfaces in the industrial domain for collaborative workshops, this paper will provide multiple applications scenarios for automotive production validation.

Presenting industrial application scenarios using augmented floor surfaces

In industry, novel products and their assembly processes have to be validated to ensure efficient production. This process is called virtual production validation. Production engineers aim to optimize the product, process and resources for the future workplaces in the automotive assembly lines, and optimize existing planning data [33]. These validations take place in collaborative workshops consisting of up to 30 people. Typical tasks are rearranging workplace layouts, the optimization of product assemblability, the reduction of overall process times, the optimization of ergonomic aspects, and the reduction of non-value adding tasks, such as walk paths (see [2, 32]). Literature refers to a similar process as the “virtual continuous improvement process” [7].

In this chapter, first the technical realization of the LHRD apparatus is described, as it is required to enable all subsequently presented use cases. Therefore, a methodology framework is introduced by the authors, which is called the “Virtual Manufacturing Station” (VMS). It consists of LHRDs and 3D tracking systems (see Fig. 1) and allows to combine the advantages of physical and virtual validation by isometric registration and tracking of all components. Second, six novel industrial use cases are presented and set into context of the design space of the above-mentioned publications.

Fig. 1

The VMS apparatus consists of two 16 sqm, L-arranged LED walls next to a 54 sqm large-scale LED floor

Augmented floor surface apparatus

The so-called virtual manufacturing station represents a multi-display environment (MDE). The apparatus consists of three LHRDs. Two identical LED walls, each sized 6.0 m × 2.7 m, are arranged in a 90^∘ L-shape with a closely attached large-scale LED floor (see Fig. 1). The specifications of the LED walls and the LED floor are shown in Table 1.

Table 1 Specifications of apparatus

Even though the setup resembles a CAVE with its three adjacent LED displays, the setup intentionally has left a gap of 120 mm between each wall and the floor. On purpose, the setup partly forgoes the immersive effect of a typical CAVE (see Cruz-Neira et al. [13]) in order to achieve maximal mechanical flexibility. At the borders of each display, standardized aluminum profile structures offer the possibility to mount additional devices, such as sensors, haptic devices, cameras, and other tracking equipment.

Within each of the two LED wall, seamless images are generated by using 20 concatenated cabinets of LED modules summing up to a 6.0 m × 2.7 m setup. Each LED wall consists of “5 × 454” Leyard TWA cabinets with an pixel pitch of 1.25 mm, resulting in an overall native resolution of 4800× 2160 pixels (see Table 1). The maximum brightness is defined to be 800 cd/sqm with a horizontal viewing angle of 160^∘. For indoor usage, brightness is set to approximately 20% intensity without daylight influence. Colors change when looking from steeper angles, due to limited horizontal field of view of the single LEDs (160^∘ horizontal viewing angle).

The LED floor consists of 216 “Uniview LED I Series” cabinets resulting in a 9.0 m × 6.0 m setup. Each cabinet has a dimension of 500 mm × 500 mm and therefore has an arrangement of 18 by 12 cabinets for the whole floor (see Table 1). Dalton et al. found in their paper that “pixel density, over the range of tests, is less important than visual artifacts introduced by carpet tile edges” [15]. Carpet tile edges can hardly be seen for the concatenated floor tiles in this setup. These seamless tiles also offer interactive direct touch capability with optical proximity sensors for floor step detection as depicted in Fig. 3. It has 16 sensors in each floor module with a latency of 10 ms. As practical design considerations, the LED floor tiles are water-resistant IP65, have an anti-scratch coating, can be replaced without additional adjustments, and can carry point loads up to 2000 kg/sqm. For automotive production validation, the latter specification is required for mixed reality assessments, putting a physical car body as a mock-up in the VMS and registering it to the virtual scene.

Interactive walk path optimization

Assembly workplace layouts have to be optimized with respect to the product, process, worker, ergonomic aspects, and the reduction of non-value adding tasks, such as walk paths. As already proposed by Otto et al. [33] in 2014, an augmented floor visualization system can be used for walk path validations and optimizations. When planning and validating assembly workplaces in the VMS, station layouts have to be modified by production engineers, so that the simulated walk paths are reduced, since they are not value adding to the product. Drift situations in assembly flow lines are crucial to be assessed, as they degrade the overall efficiency as well. Therefore, virtual simulation tools are applied and work place layouts are virtually generated for variant-rich simulations.

Large-scale augmented floor surfaces display the bird’s eye view of these virtual work place layouts and the corresponding simulation results as depicted in Fig. 2. Participants interactively optimize and validate these generated results.

Fig. 2

Walk path optimizations on the augmented floor surface. A combination of automatically simulated walk paths and interactively recorded walk paths are shown in true to scale

Being one of the co-authors, these optimization tasks are described in the paper Agethen et al. [1]. Walk paths can be optimized by visualizing (see Fig. 2 right) the automatically simulated walk paths (see Fig. 2 left). These simulated walk paths stem from a dedicated motion simulation framework [1] and are displayed on the augmented floor surface in true to scale. Subsequently, a user validates the walk path simulation’s outcome by means of re-enacting the walking tasks. Actually carried out walk paths can be displayed as heat maps showing parameters of the actually captured motions, e.g., visualizing speeds, process flows, and times.

The presented approach can be regarded as a closed feedback-loop between dynamic simulation and the real user’s movement. As the synthesized and the user’s motions are directly compared, invalid simulation outcomes can be detected at an earlier stage. This increases the maturity of planning data. Additionally, the automatic simulation framework can generate process-variants or predict the impact of different parameters (e.g., the height of a person or the weight of a part) according to the acting user’s motion.

Consequently, assembly tasks being performed on the augmented floor surface (see Fig. 3) can be further enriched using synthesized data. Therefore, it is possible to cover a wide range of process-variants in the overall assembly process simulation, while ensuring a high degree of realism by means of comparing the simulation model with the captured walk paths. Setting this application scenario in context with the design space ideas of Vermeulen et al. [44], the user’s captured positions get visualized as a “halo” on the heatmap. Walk paths are trails with “historic information” of their actions.

Fig. 3

Optical sensors of the augmented floor surface are able to detect objects and foot steps. Walk path trajectories can be derived using this sensor information

Layout assessments

Similarly to the use case of walk path optimization, virtual work place layouts are applied to optimize the overall arrangement of resources (i.e., racks, carriers, AGVs) and product parts in a work place, not only limiting the scope to walk paths. This implies also checking the availability of all resources, overall fluent processes, and process robustness (see Fig. 4).

Fig. 4

Bird’s eye view on a virtual assembly work place layout. The content is displayed on the augmented floor surface and presented in true to scale to multiple people. White and blue boxes represent simplified outlines of carriers, racks and AGVs

Therefore, the augmented floor surface helps to display true to scale virtual station layouts visualized in a bird’s eye view. Process flows and variants of the process are augmented on the floor, so that all participants geometrically can assess product, processes, and tasks inside a work place. As the workshop participants share the same CVE, they are enabled to discuss more profoundly about the work place layouts.

Virtual work place layouts consist of simplified representations, so that racks and carriers are reduced to 2D or 3D boxes with text labels. Using an orthographic bird’s eye view on the augmented floor surface, even complex virtual 3D environments are reduced to 2D projections.

Virtual true to scale stencil

Based on the above-mentioned virtual station layouts in production validation, occasionally these virtual layouts have to be built in real-life physical mock-ups, e.g., as a cardboard PMU. Serving this use case, the LED floor can be used as a virtual stencil.

True to scale representations of virtual contents help to physically replicate the work place and to validate layout variants with the racks, carriers in the automotive assembly line. Figure 5 shows the interaction cycle for both digitizing a work place and the hardware realization in the physical domain. No rulers and no protractors have to be used to arrange the physical items, as the augmented floor surface functions as a virtual stencil.

Fig. 5

Block diagram for the use case station layout planning. Both the physical and virtual domain bidirectionally influence each other. Left: physical cardboard workshops make use of haptic materials and real life forces. Right: In the virtual domain station, layouts can be quickly altered and simulated

Having physical items present, such as carriers, racks, or tools, they can be scanned, remodeled, and tracked again. This closes the loop from the physical to the virtual domain. This represents the idea of a digital twin in Industry 4.0.

Having also tracked physical items, the augmented floor surface is able to visualize its virtual meta-information, such as contents inside of a rack, dimensions of a carrier, or simulation results for walk paths. These findings are in accordance with the concepts presented by Müller et al. in BaseLase [31].

Size perception and engagement

As production validation engineers make use of these application scenarios on a daily basis, they have to be able to estimate sizes properly. In contrast to relatively scaled visualizations, the augmented floor surface helps to identify problems with clearances and other geometric details of virtual station layouts when showing it in true to scale. Rogers et al. found that the arrangement of output devices has a huge influence on the process of idea generation and discussions [38]. Exactly this effect is leveraged by using an augmented floor. Also, size perception of true to scale contents is rumored to be more precise and more accurate. This is evaluated in the latter of this paper.

Self-navigation in virtual space

Similarly to the concepts presented by Gugenheimer et al. [22], the augmented floor surface helps to opt-in and opt-out in the virtual scene. Registering all display devices isometrically to the virtual scene, production validation engineers can easily see the virtual borders of their region of interest. For example, when assembling the rear back light of a car, the interactively tracked user has to get a reference, where he is located in the virtual domain. Seeing the car from the bird’s eye view on the augmented floor surface, one can easily walk to the region of interest and perform the assembly task there. Even when using a VR head mounted display, the user can walk to the respective region of interest and puts on the VR goggles when being already perfectly aligned with the virtual domain. Vermeulen et al. propose “Halos” for self-navigation [44] and visualize borders of the tracking environment on the floor as well. This feature helps to simplify interaction in virtual domain.

Virtual travels with interactive maps

Planning new work contents and tasks for an existing work place requires changes in the respective work station (brown-field adjustments). So it is necessary to get a clear understanding of the current baseline situation. Just like “google street view” photospheres for outdoor situations, current indoor workplaces are scanned as 360^∘ photospheres and 3D point clouds in defined intervals. Within production validation, this data is used to take a look at remotely located factories all around the world.

As depicted in Fig. 6, the workshop participants see the 360^∘ images of the distant workplaces and the map at the same time. The two L-shaped walls are showing two perfectly stitched images. The virtual cameras always keep a 90^∘ offset around the vertical axis. Additionally, the augmented floor visualizes a circular map of the factory perfectly aligned with the heading of the viewport of the two walls. This allows the user to get a perfect overview, where you are currently located and in which direction you currently look at. They dive into real-world scans and understand the circumstances directly. They can walk through the distant factory, look around and measure directly in 3D space.

Fig. 6

Usage of the augmented floor surface for virtual travels. Left: The schematic shows the information architecture on both walls and the floor. Right: The walls allow for an immersive deep-dive in 360^∘ photospheres, each virtual camera having a 90^∘ offset around the vertical axis. The floor map orientation changes accordingly to the wall viewport

As this LHRD setup is utilized by production engineers on a daily basis for the above-mentioned use cases, the question arises, whether they really get better in their personal size estimations and whether they get faster using these tools provided by the VMS. For validation purposes, people have to estimate sizes as precisely and accurately as possible in order to judge the validity of planning data. Even when utilizing complex 3D models in combination with an orthographic, non-tracked virtual camera, visualization contents, such as racks and carriers. Czerwinski et al. also indicate that LHRDs have an influence on productivity and satisfaction [14]. Therefore, the following chapter presents an size perception study using 2D content representations, just like in the aforementioned use case “layout assessments.”

True to scale size perception study

This study focuses on how people perceive sizes of virtual contents using immersive display technologies with true to scale data visualization. Therefore, size judgment performance is compared between three scenarios, two of them showing to scale data representations on a LED floor and one showing relative-sized visualizations on a tablet computer.

Nevertheless, using LHRD technology human perception has not been in basic research focus extensively. Bezerianos et al. presented a call for research [8] on the perception of data on wall-sized displays, as there is still little research carried out in this domain: “We do not yet know how the perceptual affordances of a wall, such as the wide viewing angles they cover, affect how data is perceived and comprehended” and call “for more studies on the perception of data on wall-sized displays.” Using different types of display devices directly influences the spatial perception, visual space, and the control of spatial behavior, especially when using display arrangements such as a LED floor. We follow this call for research and add an additional element: Data visualization on large scale augmented floor surfaces showing contents in absolute scale.

In the purely physical domain, size and distance judgments have been in the focus of literature for a long time. In 1963, Epstein [17] presented the key findings that distance and size judgments are not systematically related and deviations of size judgments varied with distance. Later, Epstein and Broota [18] presented a further evaluation on the judgment of sizes and distances and the corresponding reaction times. They found a positive correlation between viewing distance of objects and the reaction time. In Wagner’s publication, “The metric of visual space” [46], he gives insights into judging distances, angles, and areas as conducted in this study. Cleveland and McGill present groundbreaking works in the visual decoding of information, namely graphical perception. They present a set of elementary perceptual tasks working and how people extract quantitative information[12]. More recently, Talbot et al. pick up these works and analyze the reasons for the differences in perception of charts [42].

For virtual environments, broad research is carried out on perceived spaces in VR, such as distances, sizes, speeds, and spaces. Loomis et al. showed that egocentric distance judgments in physical environments nearly match 100% of the actual distance [29], whereas in virtual environments, they are frequently underestimated. Renner et al. presented a literature review and summarized that a “mean estimation of egocentric distances in virtual environments of about 74%” [37]. Renner et al. also clustered possible influence factors for this under perception of sizes in four different clusters: measurement methods, technical factors, compositional factors, and human factors. In contrast, current state-of-the-art head mounted displays seem to ameliorate these effects [24]. Kelly et al. showed when using modern HMD devices, this effect is reduced but has not been completely resolved. In comparison with the literature, no relative size judgment has been carried out in VR by providing the user’s with relative scales.

To the best of our knowledge, we are the first to execute size judgment experiments using a large-scale LED floor setup in comparison with a small-sized baseline measurement.

Study goal and predictions

One of the striking benefits of a large-scale displays is the possibility of visualizing true to scale data, contents, or virtual scenes. In the context of the presented use case within the automotive industry, 3D contents with individual view points have been intentionally excluded, whereas 2D representations (see Figs. 7 and 1) have been chosen for this study, since the aforementioned use cases are limited to data visualization of 2D data.

Fig. 7

In all three scenarios, a square is shown. It is randomly scaled, positioned, and rotated. Additionally, a 1-m ruler is given as an additional visual cue

This evaluation gives insights if people can assess sizes of 2D contents more accurately and precisely if they are shown in true to scale compared with relative-scaled representations. The baseline scenario represents relative-sized visualizations on a tablet computer, showing exactly the visual cues as in the true to scale scenarios. In this study, size judgment refers to the edge length estimations. To date, there is no published research documenting the extent to which true to scale floor content supports people in estimating sizes using augmented floor surfaces. To address these issues thoroughly, this study employs verbal distance judgments and objective measurements. Four different aspects are evaluated in this study:

• Accuracy: Is there a systematic overestimation or underestimation (accuracy) of size judgments? (Mean absolute percentage error, see Armstrong and Collopoy [5])

• Precision: In which scenario participants achieve the most precise size judgments. (SD of mean absolute percentage error).

• Task completion time: Is there a difference in task completion time for the three different scenarios? (Objective time measurements)

• Qualitative feedback: Are the user’s subjective size judgments on precision and task completion time matching the objective measurements? (Non-standardized questionnaire)

Participants

For this study, 22 voluntary participants were randomly selected, such as production engineers, research engineers, PhD candidates, and students from different production planning departments in manufacturing industry. Fifteen males and 7 females were taking part, all ranging from 21 to 57 years. (M= 31.57, SD = 11.52). All participants reported normal to corrected vision and chose the metric system as their preferred unit.

Setup, stimuli, and design

Three different modes of perception are evaluated. For all three scenarios, the same visualization software, visual cues, and interaction (besides user’s movement) are used, only the output modality is changed (see Fig. 8):

• Tablet scenario (T): Relative-sized visualizations as a baseline

• Floor scenario (F): True to scale visualization restricting user’s viewpoint on the side of the LED floor

• Floor and Interaction scenario (FI): True to scale visualization allows user’s movement on the whole LED floor

Fig. 8

Three evaluation scenarios: Tablet(T), Floor(F), and Floor Interaction(FI). Left: The user carries out the size estimations using a tablet computer. Center: The user utilizes the augmented floor surface, standing on the outside. Right: The user moves on the floor while performing the size estimations

The rendering and evaluation software is a custom application which displays virtual squares in a randomized order (six different sequences for 3 scenarios) handling the randomized scenario work flow and logging the evaluation results (square size, square rotation, pixel per meter, scenario completion time). In all three scenarios, the participants are shown 2D white squares on a black background. These squares have randomized sizes from 50 to 200 cm with random positions and orientations (+/–15^∘) on the screen (see Fig. 7). Additionally, a virtual ruler represents the absolute length of 1 m and remains at the same position (center bottom) throughout all scenarios. Besides the aforementioned 9-m × 6-m LED floor apparatus with 10.81-m screen diagonal for the scenarios (F) and (FI), scenario (T) is visualized on a 12,3” tablet screen, set to the same aspect ratio as the LED floor. The LED floor pixel pitch is 5 mm.

Procedure

After signing the informed consent, the participant is given verbal instructions on the goal and evaluation procedure. Each participant executes all three scenarios (T), (F), and (FI) (within-subject design) in a randomized order to abolish learning effects. There is no interaction with the virtual contents, so that the focus is limited to the differences in spatial perception. In each scenario, 20 randomized (size, rotation, position) squares are visualized. After presenting each square, the participants verbally express their size estimate to the experimenter in the unit centimeters. The experimenter writes down the response for each estimation in parallel.

The three different scenarios are depicted in Fig. 8 and described as follows:

• Tablet (T): The software visualizes the squares on the tablet computer as relatively sized content. The users have to judge the absolute edge length in relation to the visualized ruler.

• Floor (F): The software visualizes the squares on the LED floor to scale. The participant is directly facing the LED floor from a static location (compare [24]), standing on the outside border, centered on the long edge of the LED floor (3m to the center), and may not access it.

• Floor&Interaction (FI): Same setup as in scenario (F), but in contrast, he has the opportunity to move freely on the augmented floor during the study, so that the subject may position himself/herself directly above the respective square.

The experiment has been conducted a total of 22 times with different participants. Each evaluation takes approximately 20 min including the subsequent completion of the questionnaire. A total of 1320 datasets have been collected (22 participants, 3 scenarios, 20 trials) each one containing the actual and reported length [cm], spatial deviation/error [cm], task completion time [ms], pseudonym, scenario, square rotation, and position.

Finally, participants are handed out and asked to fill out a questionnaire after execution of all three scenarios to gather their subjective feedback. They are asked about their personal scenario preferences for direct comparison. In addition, each subject is to select the method which he/she has preferred and specify the reason for his decision.

Results

The results are clustered in the three sections: Accuracy, precision, and task completion time. Spatial deviation is the difference between the actual edge length (ground truth) of the squares and the estimation of each participant for the respective square edge length. Negative values represent an underestimation of size and vice versa.

Figure 9 shows an scatter plot of all three scenarios depicting the true length [cm] over the difference between true and estimated length. All three scenarios show, that in mean, there is only little overall overestimation or underestimation of the user’s size judgments with (T) having a mean of 0.951 cm (SD = 30.204), (F) − 0.634 cm (SD = 22.499), and (FI) − 5.694 cm (SD = 17.850). However, regarding the relatively large standard deviations compared with the small means, the interpretability of the aforementioned spatial deviation is disputable due to overestimation and underestimation. Furthermore, by tendency, the spatial deviation rises with growing edge length of the squares, especially considering (T) and (F). In order to normalize these effects, in the following, the mean absolute percentage error (MAPE) and mean standard deviation (SD) of MAPE for trials within subject are used to evaluate accuracy and precision between all three scenarios.

Fig. 9

Scatter plots of all three scenarios show the spatial deviations. Each plot is following Bland-Altman plot [10] style, additionally showing the mean values, standard deviations, and a linear regressions over the actual visualized cube sizes. By tendency, one can see an increase of variance of spatial deviations over the true sizes in all scenarios

Accuracy

MAPE is a measure of prediction accuracy. (T) shows a mean absolute percentage error of 14.783% (SD = 5.612%), (F) 11.369% (SD = 4.599%), and (FI) 9.814% (SD = 3.957%). Figure 10 depicts the box plots of the MAPE of all three scenarios. A statistically comparison is conducted considering (T), (F), and (FI). Levene’s test shows that variance homogeneity is given for this data (F(2,63) = 0.942, p = 0.395); therefore, the standard one-way ANOVA can be used in the latter. One-way ANOVA reports statistically significant difference between the three scenarios (F(2,63) = 6.242, p = 0.003). The post hoc pairwise t test with Holm correction reveals that there is no significant difference between (FI) and (F) (p = 0.284), but for both other scenarios (T) and (F) (p = 0.041) and (T) and (FI) (p = 0.003).

Fig. 10

Box plot for MAPE of scenarios (T), (F), and (FI)

Overall, therefore, the MAPE of both true to scale visualization scenarios (F) and (FI) can be regarded as significantly different from the relative scaled (T) scenario. As both mean MAPE values are lower, the scenarios (F) and (FI) have a higher accuracy compared with (T).

Precision

The mean SD of MAPE for trials within subject demonstrates the precision of size judgments represented by the “variance of absolute percentage errors.” (T) shows a mean SD of 10.006% (SD = 3.394%), (F) of 9.759% (SD = 6.051%), and (FI) of 8.921% (SD = 7.898%). Figure 11 depicts the SD of MAPE for trials within subject box plots of all three scenarios. Levene’s test is utilized for testing equality of the variances in distributions. With F(2,63) = 0.329, p = 0.721 it shows that variance homogeneity is given for the SD. Therefore, standard-one way ANOVA with post hoc pairwise t test with Holm correction can be used in this case which reports F(2,63) = 0.184, p = 0.832. Since one-way ANOVA shows no significance, post hoc test results are not reported here.

Fig. 11

Box plot for SD of unsigned percentage errors in scenarios (T), (F), and (FI)

No significant difference in precision can be found using true to scale visualization scenarios (F) and (FI) compared with (T). However, considering the descriptive statistics of mean SD of MAPE for trials within subject, a minor tendency of lower precision of (T) compared with (F) and (FI) is depicted (see Fig. 11).

Task completion time

The participants did neither get any instructions on task execution time nor on the priority between precision and speed. Nevertheless, task completion time has been tracked throughout the experiment. Time measurements have been gathered for every single size estimation in all scenarios, stating when a square is displayed and finishing when verbally passing the size judgment to the study manager.

Participants show a training curve throughout the 20 runs of each scenario. All in all, run 2 to 20, the median of scenario (T) is 5.063 ms, whereas the scenarios (FI) (9.959 ms) and (F) (8.429 ms) are slower. For all three scenarios, the very first runs show a higher median values (see Fig. 12 and Table 2) caused by non-existing training.

Fig. 12

Median task completion time of all participants (N = 22) for all 3 scenarios throughout the 20 runs

Table 2 Comparison of median task completion times

Questionnaire results

After having performed the experiment, all 22 participants filled out a questionnaire on their subjective perception. The non-standardized questionnaire compares the objective metrics with the participant’s subjective perception.

Task completion time

“For this method, I was able to judge the sizes more quickly.” The participants had to decide on each possible pairwise combination of all three scenarios: “(T) or (FI),” “(T) or (F),” “(F) or (FI).” Overall, the subjectively fastest scenario is (T). Comparing the scenarios (F) and (FI), the results are equal (50% vs. 50%). Comparing both floor scenarios (F) and (FI) with the (T) scenario, a subjective time benefit of (T) is reported 72.73% in favor of (T) compared with (FI) and 63% in favor of (T) compared with (F). The subjective questionnaire feedback matches the objectively measured times. 86.67% of the participants were really quicker, when they are in favor of the (T) scenario in terms of task completion time. In contrast to that, only 7.14% of people in favor of (F) or (FI) scenarios were really quicker.

Precision

“Using this scenario, I’m able to assess the sizes more precisely.” As for task completion time, all pairwise combinations of scenarios are tested: (FI) is estimated the most precise scenario (46.97%) followed by (T) (31.82%) and (F) (21.21%). Interestingly, people clearly preferred (FI) over (F) (86.36%), whereas when comparing (FI) with (T) and (F) with (T), there is no clear preference (50.00% and 54.55% in favor of both floor scenarios). Comparing those subjective results with objective error metrics, there is a false impression for the subject’s error estimation capability using (T) scenario. Only 28.57% objectively performed more precisely using (T) even though they are estimating this scenario as the most precise one. In contrast, 78.26% of people who are in favor of either (F) or (FI) scenarios also objectively performed better using these scenarios. Additionally, participants reported on their absolute subjective size judgment error. In general, participants objectively performed better with a lower absolute median error than they subjectively expected it to be (positive values only) (see Fig. 13). For (T) scenario, the perceived median absolute error is 20.00 cm, whereas objective median error is 14.08 cm. The same holds for (F) (perceived 20.00 cm, objective 11.08 cm) and (FI) (perceived 15.00 cm, objective 9.25 cm)

Fig. 13

Comparison between perceived (subjective) absolute spatial deviation and objective absolute spatial deviation. N = 22 for each scenario

Personal preference

“I personally prefer the following scenario”: The highest ranked scenario is (FI) with 59.09%, followed by (T) (31.82%) and (F) (9.09%). Despite (T) is ranked second as a preferred scenario, participants who preferred this scenario never performed best (0/7) in terms of precision and most of them even performed the worst (5/7). Additionally, the questionnaire gathered free answer possibilities: The participants reported that when using (FI), they felt “more confident estimating sizes” (3×), “used natural walking” (1×) to estimate the absolute lengths and to change their “viewing perspective” (2×) so that the squares are “right in front of them” (1×). They report to get a better “spatial sense” (1×) and realism degree (2×). Additionally such a true to scale visualization is helpful. People who prefer the (T) scenario subjectively mentioned a better “overview” (3×) and better “comparison with ruler” (2×) due to the smaller display size and “higher resolution” (1×).

Discussion

The results of this study indicate that both absolute- (true to scale) and relative-scale visualizations have advantages:

For absolute-scale visualizations, there is a significant change in size judgment accuracy between tablet and both floor scenarios (F) and (FI). Using the LED floor with true to scale visualization has a positive influence on the precision of size perception. These experimental results are in accordance with earlier findings by the authors (see Otto et al. [33]). There, cascadable, room-scale projection systems are used to realize also industrial applications. In addition to LED-based and projection-based systems, more and more industrial application scenarios are realized using VR/AR interaction techniques. Using these head-mounted displays (HMDs) lacks two main benefits compared with augmented floor surfaces: First, HMDs are single user devices whereas augmented floor surfaces can be utilized with groups up to 30 people. Second, perceived spaces and sizes are frequently underestimated using VR HMDs (following Kelly et al. [24]) even though this effect gets smaller with state-of-the-art headsets. In contrast, LED floors do not show effects of overestimation or underestimation following the results of this publication. Therefore, using true to scale visualization enables the participants to judge sizes more accurately.

For relative-scale visualizations, task completion times tend to be lower. Overall, using the scenarios (F) and (FI) is slower than using (T). Even though, lower task completion times could be a hindrance factor for other use cases, in automotive production validation, task completion time is less important than a high accuracy.

Another interesting effect in human size judgments are rounding habits: All participant reports size judgments in a rounded form: Typical reports of size estimation granularity are 5 cm (5/22), 10 cm (16/22), and 25 cm (1/22) steps. None of the participants gave sub-centimeter precision results. Therefore, rounding effects are still smaller than the perceived size judgment capability (compare Fig. 13).

Conclusion and outlook

So far literature did not provide sufficient publications on the industrial usage of augmented floor surfaces. Therefore, this publication first introduces the application of large-scale floor displays for true to scale automotive applications. Six novel application scenarios in the domain of virtual assembly validation are proposed:

• Interactive optimization of walk paths

• Layout assessments

• Virtual to scale stencil

• Size perception and engagement

• Self-navigation in virtual space

• Virtual travels with interactive maps

With these application scenarios, researchers, practitioners, and production engineers are able to adapt these use cases to their own needs and realize similar systems in a broader range of industrial use cases. As the proposed hardware arrangement and the application scenarios are used on a daily basis, the VMS methodology proofed to deliver good results in collaborative workshops for production validation in automotive industry.

Furthermore, this publication provides the readers with in-depth insights into human size judgments and task completion times, as all of the above-mentioned applications rely on accurate and precise user’s size judgments, different modes of perception, and faster task completion times. Comparing true to scale (absolute) and relative scale visualizations, size judgment accuracy is better using absolute visualization scenarios (F) and (FI), whereas task completion time rises using those scenarios compared with the baseline scenario (T). In comparison with VR spatial estimations, where sizes are frequently underestimated, for true to scale floor visualizations, no generalizable deviations could be revealed. Various use cases depend on reliable spatial estimations of humans, such as collaborative production validation workshops. The presented apparatus consists of an 54-sqm LED floor with a 5-mm pixel pitch and proofed to be a helpful tool for visualization of virtual true to scale contents.

Future research will focus on interaction in isometrically, co-located virtual environments. For this purpose, true to scale visualizations both on the LED floor and LED walls are an enabling technology. Additionally, using the presented apparatus, further interaction research will be carried out as it offers the possibility of optical foot step recognition. Human walk paths can be reconstructed and recorded trajectories can be directly compared with simulated ones.