1 Introduction

Traditional product development involves many stages, which address crucial design decisions. The most important method to acquire user feedback, communicate core design concepts, and compare different designs is prototyping [1]. In the context of industrial product design, e.g. automotive industry (cf. Fig. 1), mobile machinery (cf. Fig. 2) or architecture [2], usually physical prototypes are created to visualize a specific aspect of the current concept. Due to the increasing availability of process information, products for professional applications such as agricultural machinery, e.g. harvesters or tractors, contain an increasing number of displays, touch screens and controls. Traditional physical prototypes are not capable of reproducing the functionality of these virtual elements and thus it is difficult to evaluate the interaction between product and user interface. Integrating virtual elements in prototypes such as clay models is costly and time-consuming. Moreover, the limited flexibility of the prototype itself renders this tool unusable for quickly testing a design idea or evaluating a rough concept at an early design stage. Virtual Reality (VR) technologies can aid in assessing the interplay of devices, controls, and displays in the product under development. In general, two different approaches are possible:

Fig. 1.
figure 1

Visualization for a HUD-concept and gesture-based control in an automotive context.

Full size image

  • augmentation of physical prototypes with (interactive) virtual content

  • virtual simulations of physical prototypes

Due to the current technological advancement, virtual prototypes are affordable, quickly realized and easy to modify. Hence, these prototypes provide an option to overcome the deficits of physical prototypes, which are typically created when the design is already quite mature.

In the next section, we present related work and afterwards discuss virtual prototyping in more detail. We then address research questions and propose a set of guidelines including interaction and visualization techniques to overcome current limitations. Moreover, we address practical issues we experienced during the development of prototypes in the context of agricultural machinery and automotive industry. We discuss the integration into the design workflow and conclude with future work.

2 Smart Prototyping

Digital modelling, simulation and communication has long been an integral part of product development. VR technologies and especially Virtual Prototyping can be used to define design specifications [3] or as methods for participatory design [4]. VR has also been investigated as a design tool for quick evaluations [2, 5]. Hence, users can evaluate the usability of product interfaces using VR, which acts as a means to communicate design decisions and alternatives. Moreover, virtual training scenarios allow users to experience a tool, machine, or process without the need to stop operations of devices and factories or risking health threats [6, 7]. Heydarian et al. worked on a study comparing immersive virtual environments with their physical equivalent focusing on the user performance in test activities. The results show that the participants’ performance was only slightly influenced by the type of the environment and they also felt a strong sense of presence within an interactive Virtual Environment [8].

Although no complete and evaluated approaches to virtual prototyping have been proposed, previous research projects provide valuable starting points that show how to proceed in this area. For instance, Frund et al. use augmented reality as a design tool in order to blend virtual objects such as 3D models or user interfaces with partial physical prototypes [9]. Zwolinski et al. focus on early design stages, using VR technologies to define basic product structures based on functional requirements [10]. Architectural visualization has already been investigated as a field of application for virtual prototyping, involving users in the planning process of a participatory design process [11, 12].

Using VR to present early design concepts offers several opportunities to aid the participatory design process. The communication between users and designers is significantly enhanced with interactive prototypes [13]. Ideas and concepts are easier to understand if the user can experience the intended effects of the design interactively. Another idea behind virtual prototypes is to constantly gather feedback from users during the whole design process by using VR prototypes containing only elemental parts to visualize a design concept. When the design and the entire product become more mature, existing VR models can be extended to facilitate an in-depth evaluation. During our interdisciplinary research, we identified another advantage: VR helps to communicate the different requirements and priorities of participating disciplines and boosts the collaboration within the development team by providing an easily comprehensible and dynamic form of visual communication.

For a long time, the immersive presentation of VR content has been very costly, involving large power walls or CAVE settings. With the introduction of lightweight and powerful consumer electronics such as current Head-Mounted-Displays (e.g. Oculus Rift [14]) or smartphone-based VR headsets (e.g. Samsung Gear VR [15]), VR Prototyping and the immersive presentation becomes a reality for a wide range of developers and end-users. This addresses another requirement for virtual prototyping: portability. Using smartphones in combination with Google Cardboards [16] as low level VR Experience proved to be very effective for communicating ideas to project partners, visualize concepts or simply exchange ideas. This represents a huge advantage for the participatory design process: Instead of evaluating design concepts in the laboratory, a variety of users can be equipped with portable VR-prototypes for evaluation in a familiar environment and generate feedback directly from the potential customer.

3 Research Challenges

Apart from the visualization a main challenge of VR prototyping is the interaction with virtual models and interfaces. In the next sections we discuss the importance of multimodal approaches and several interaction prototypes, which we developed to study their suitability for different application scenarios and their effects on the user’s perception.

3.1 Multimodality

The visualization of 3D models for use with VR is still not a straightforward task but solutions for most issues are available (cf. Sect. 5). Including other modalities remains problematic. Sound represents a minor issue, although we experienced that the design workflow seems to be very visual, and therefore it is quite difficult to obtain authentic sounds for a VR simulation. This situation may be specific to the scenario of agricultural machinery where sound design is not a core aspect of the design process. With haptics, the situation is different: Current solutions to simulate haptic sensation are expensive, complex, and not very portable.

As VR prototypes focus on simulating only certain aspects of the design concept, especially for evaluating the visualization or interaction design with touch interfaces, missing haptics and other modalities may be acceptable. For evaluation of physical controls, physical prototypes seem to remain the most feasible option, however VR can be used to combine these physical controls with virtual elements, e.g. for information visualization aspects. Another approach goes one step further by replacing concrete physical objects with dummy objects, which incorporate a similar, but not necessarily identical shape and materiality in order to simulate these aspects with VR visualizations. The question remains, whether the optical illusion of authentic shape, texture and material is sufficient to trick the human perception.

3.2 Interaction with VR Prototypes

Besides the difficulties to simulate certain modalities, interaction with VR content represents another major issue. For our idea of interactive prototypes, we concentrate on basic tasks, such as (guided) navigation, selection and basic manipulation tasks. As we target an early design phase, exact simulation of complex interactions is not necessary, especially when evaluating visualization parameters or the spatial layout of interactive elements in the cockpit as well as identifying opacity or occlusion issues typically found in Augmented Reality (AR) scenarios.

Fig. 2.
figure 2

Head-based interaction in context of mobile machinery.

Full size image

Especially for portable systems, options for interaction are spare. Zhu et al. investigate head movement and eye tracking as natural forms of interaction [17]. The authors show that head movements can be more easily tracked and do not cause the Midas touch problem, which is present when using eye movement as a means for interaction. Eye movement is primarily needed to investigate the surroundings, which can be in conflict with interaction. However, head movement as a tool to interact imposes greater physical strain on the user. In any cases of eye and head tracking, applying a dwell time before triggering actions in the virtual world is necessary in order to avoid unintentional actions [18, 19].

One of our prototypes demonstrates the feasibility of this approach for simple navigation tasks (entering a vehicle, moving around based on leaning forward/backwards/to the side) and interacting with control elements inside the cockpit of the machine (cf. Fig. 2). Even basic steering actions of the vehicle could be performed, although reaction was very high and accuracy quite low. This illustrates the core problem of dwell-time based interactions: it is simply not suitable for time-critical interaction steps.

Gesture-based interaction seems another option for more or less portable VR systems. However, in VR the absence of the user’s body represents a serious issue for this type of interaction. The integration of an egocentric avatar in the virtual environment represents a possible solution and offers several additional advantages. Due to representation of the user’s body and knowledge about real world scales, the predictability of size relations and virtual distances can be improved [20, 21] which is highly relevant for virtual prototyping. In addition, it can be assumed that the depth of presence increases in Virtual Reality applications, if the body of the user operates according to his real world movements [22]. A strong sense of presence is a desirable feature for virtual prototyping, as you want the user to perceive the model and world as natural as possible.

4 Virtual Reality Prototyping Guidelines

Especially interaction techniques for prototypical VR experiences are an open research field. We present several approaches, realized in the depicted examples. Furthermore, the feasibility of current 3D frameworks to quickly create a reasonable VR experience and their integration in the design workflow/toolchain are discussed.

4.1 Interaction Techniques

One goal of virtual reality prototyping is the substitution of physical prototypes by using virtual models [3] in order to reduce efforts in building complex functional physical prototypes. Using VR or AR techniques, they therefore can be realised much cheaper and in early phases of the development process. This enables a better integrated evaluation and participation when designing new products. The layout of interfaces elements such as steering wheels, pedals, monitors, buttons etc. can be evaluated by using the VR simulation. We propose a wide range of interaction techniques which enable the utilization of different interaction techniques depending on the specific needs of the simulation. Available options in our prototypes are:

  • gaze-based interaction, simulated by head movements

  • leaning-based interaction, using torso movements

  • use of gestures, especially hand and finger movements

  • interaction techniques using existing devices such as steering wheels

To address specific needs of virtual prototyping applications we present an overview of the mentioned interaction techniques and the corresponding suitability to different tasks and requirements according to our experience (cf. Table 1).

Table 1. Interaction techniques and the suitability to different tasks and requirements: ++: excellent, +: good, ○: neutral, −: poor.
Full size table

Gaze and head-based interaction can be used for simple, not time-critical tasks. Their main advantage is that they represent functionality built into the most devices, are applicable to most scenarios and do not require additional wiring or tracking. Gestures, and leaning as special gesture, do need additional, lightweight tracking. Leaning is suitable for navigation tasks as it is easy to learn and understand. Depending on the complexity and suitability for the associated task, gestures are more demanding both for the user and the tracking hardware. Devices are the least universal solution. However, as they are designed for one specific purpose they are unmatched in terms of ergonomics, ease of use and efficiency.

Another approach utilized in our prototypes is to reduce complexity of the interaction by mapping complex real-world interactions to simple inputs to trigger these actions in VR. Although this approach does not completely reflect how the user would interact with the according system in reality, the efficiency of visualizations, or the user interface design in general can be quickly evaluated. On the other hand, these interactive prototypes can serve as playground for designer to try concepts for novel interaction techniques and iteratively refine existing concepts.

4.2 Visualization Techniques

Still a problem at present is the artificial character of virtual avatars. We found tendencies for the intensification of mental immersion once users perceive their own individual body in virtual environment. Thus, we developed a prototypical application, which superimposes a captured video stream of the user’s current view on the virtual scene in real-time. By means of a chroma-keying segmentation method only skin tones will be displayed, such that the user is able to see his authentic hands form, colour, pigmentation and hairiness (cf. Fig. 3). [23] According to our research, the user experience can be enhanced with this system. Evaluation results show significant improvements regarding stimulation and originality scales [24]. It can be assumed that Virtual Prototyping can benefit of the increased hedonic quality and experience-oriented focus of the application. In combination with markerless tracking systems, freehand gestures will allow the user to interact with virtual objects or navigate through virtual scenes. Current technological developments even allow a portable tracking when using a small depth camera mounted at the VR headset for hand tracking purposes (cf. [25]). Therefore, we recommend the integration of an animated user avatar or the adaption of the proposed video overlay technique for virtual prototyping. In our opinion the benefits in terms of size and distance estimation, the deeper sense of presence and the increasing user experience are important key points for a successful prototyping and design process.

Fig. 3.
figure 3

Visualizing the user’s body in virtual environments (Günther, Franke & Groh 2015).

Full size image

We created several prototypes visualizing several different types of agricultural machinery and the interaction with these machines, especially different cockpit layouts and control paradigms. For a better understanding of the functionality of these complex machines, we developed interactive visualizations that show different types of harvesters. The resulting VR prototypes used different visualization techniques to reveal the technical components inside the harvester and the parameters to control the harvesting process to reveal the inner components of the machine (cf. Fig. 4, left):

Fig. 4.
figure 4

Examples of prototypes visualizing components of harvesters (left) and UI-concepts for controlling the machine (right).

Full size image

  • transparency

  • wireframes

  • dynamic masking of the machine hull (hiding unnecessary parts of a complex system based on the user interaction, position or view direction)

This example shows one advantage of interactive VR prototypes for presentation purposes and training scenarios. Existing CAD-Models can be used and enhanced by interactive components and visualization techniques to visualize functionality of a complex system and interaction between components. Furthermore, interactive prototypes can be used for customers to show the technical evolution and advantages of upcoming systems. Especially for these scenarios, we encourage the use of special capabilities found in VR, such as annotations or overlays, transparency or masking of elements, as well as stylized rendering techniques such as outlines or special shading techniques.

Prototyping innovative user interface concepts for controlling complex machinery, especially visualizing parametrisations of system components and their side effects, represents another use case. Nowadays, agricultural machinery (or at least parts of it) are often controlled using touch sensitive screens. The transition from traditional hardware-based interaction techniques towards the interaction with virtual content leaves much space for improvement of the visualization and interaction. Furthermore, AR concepts find their way into the cockpit. Although current AR technologies are quite limited, basic questions on future AR visualization techniques and solutions for typical problems such as text legibility, the amount of information which can be presented, contrast or occlusion represent typical design parameters which need to be considered.

Fig. 5.
figure 5

Simulation of AR concepts using VR.

Full size image

One approach is to use VR to simulate typical behaviours of AR visualizations (cf. Fig. 5). One benefit of this method is, that concepts can be evaluated which circumvent current technical limitations offering the opportunity to focus on the benefits which AR offers for visualizing virtual information integrated into the physical environment. By doing so it is possible to develop interaction and visualization concepts for future AR implementations. Additionally, the combination of different visualization and interaction techniques are in focus: questions like interoperability between touch control and AR visualizations can be evaluated.

4.3 Technology and Frameworks

Computer Graphics Frameworks have been evolved over the last couple of years. Modern graphics engines, which are available (and affordable), allow to create interactive 3D scenes without large programming overhead. This represents an advantage for design prototyping, because development times are shorter, the implementation is less error prone, easier to debug and, due to the use of visual scripting tools, interactive content can be created even without programming skills. This allows designers and programmers to work closer together and facilitates the interdisciplinary collaboration. Additionally, it helps to work with small iterations on the intended prototype or design solution.

In our recent prototypes, we relied mainly on the Unreal Engine 4 [26] and Unity 5 [27]. These frameworks consist of powerful editors for material editing, mesh import and customization as well as scripting functionality to easy make a static 3D scene interactive. Furthermore, they provide a convenient integration of VR components, especially in regards to the integration of VR headsets. They also grant access to an exhaustive collection of materials, models and plugins through their inbuilt stores. These assets, some of them are freely available, can be used to augment the virtual environment with additional elements to make the VR experience more authentic (e.g. foliage, landscape presets) without additional development overhead. Further advantages are a very active community which helps to solve technical issues, an extensive technical documentation which help to explore the more sophisticated features of the frameworks and continuous technical development.

Integration of common tracking devices is achieved by using existing plugins. Custom plugins for the framework can be implemented in order to support devices. Especially with regards to the Unreal Engine, the opportunity to integrate C++ code offers the opportunity to reuse existing frameworks by writing wrappers that grant access to the core functionality to be used by the engine. Existing state-of-the-art libraries like OpenCV [28] or ARToolkit [29] can be integrated in this way.

5 Integration into the Design Workflow

To face the complex HMI design development and the urge to exploit the advantages of evaluation and decision-making in early stages of design and development processes, frameworks for rapid prototyping are needed, which allow frequent changes of content, easy adaption of UI-designs or altering the interaction design. Additionally, VR prototyping frameworks should provide a seamless integration into the design workflow, one area which is currently still problematic, due to different technological approaches of CAD-Software and Real-Time Systems, and incompatible file formats or incomplete interchange formats. One core issue represents scaling down the existing CAD models for use with real-time systems. As CAD is often based in curves and the goal is to create as exact curvature as possible, a straightforward tessellation creates a very high polygon count. This makes it impossible to render the content with high frame rates that are needed for VR. Simple reduction is possible, but sometimes creates artefacts, as specific parts of a model may require a higher level of detail whereas other parts can be reduced even further. Reducing the level of detail for use in VR is therefore work-intensive and requires a clear communication by the VR experts about constraints regarding the rendering capabilities.

Another issue relates to the rendering of the materials. CAD software uses different mathematical models for rendering materials (mostly due to the missing real-time requirement) and therefore a different parameterization for creating a specific look. Especially when using the Unreal Engine 4, its in-built Physically-Based Material approach makes it necessary to recreate the materials after importing the model. Although the editor offers a WYSIWYG experience, material networks for authentic materials can become quite complex. A related issue is the number of possible materials, which is typically constrained in a real-time 3D scene to keep performance acceptable. Depending on the interchange format and software used, duplicate materials or large amounts of material with slightly different parameters require an additional overhaul of imported models or an additional revision of the source materials before exporting.

Rapid prototyping works for creating a more or less authentic VR experience, however, rendering performance, especially for high resolution rendering used with the oculus rift requires powerful hardware or a reduction of detail when used with less powerful hardware (e.g. mobile rendering for use with Google Cardboard). However, the code created by rapid prototyping tool provided by the editor is not optimized for performance. Additionally, the current frameworks need to mature to become more stable and deliver more predictable rendering results. Especially the Unreal Engine 4, which is under active development with rapidly changing and improving software versions, is still not very stable and suffers from performance problems especially when exporting to mobile devices.

6 Future Work

In the early design phase, a core requirement is a context-based, individual and flexible simulation. In contrast, the demands on realistic simulation and completeness of the environment are lower than for other scenarios, such as training or in-depth evaluation. Current graphics engines make this less work-intensive, however, making a VR scene interactive is still a demanding task. Especially when using proprietary tracking software or hardware, the translation between tracking result to intended interaction is tedious and error prone. There is a need for abstract interfaces between tracking hardware and VR software and for a general human interaction model. Another related issue is represented by the integration of traditional UI-Interaction (e.g. touch panels in a harvester) in a VR scene. Design concepts and UI-prototypes often use standard UI technologies such as HTML5. Integrating these prototypes in a VR experience is currently simply not possible or requires a complete reimplementation using UI-frameworks suitable for VR. This issue – integrating 2D UI in 3D scene – remains unsolved for a long time. Additional work is needed to find methods to integrate or convert user interfaces created classic UI-technologies into 3D graphics engines without breaking the complete implementation.

Interaction with VR content remains another critical issue. In some use cases, current technologies are at least usable – e.g. if the goal is to determine the most convenient spatial layout of controls or UI-elements. However, for usability evaluations or similar demanding tasks incorporating a concrete interaction of the user with the system, current systems and interaction are simply not accurate enough, are too inflexible or do not provide adequate feedback.

To explore the described technologies and guidelines as well as their potentials and usability in development based prototyping, several projects are planned. An interdisciplinary project discusses innovative approaches for future cabins for excavators. In cooperation with an industry partner, the goal is to collaboratively design a new cabin or vital components using digital and physical prototypes for testing evaluation and communication. Another project deals with the location of different types of digital enhanced prototypes within the different phases of the design process. Therefore, a wide range of technologies and forms of enhancement will be contrasted with the demands, settings and capabilities of crucial milestones within the process.