1. Introduction

This manuscript was derived from observations made by the author as part of a keynote address (‘Titan talk’) at the 2009 Winter Simulation Conference in Austin, Texas. As such, the work covers mainly research on the topic of human–model interaction from the laboratory at the University of Florida. The purpose of the paper is to overview this previous research and to emphasize the area's importance for the future of simulation modelling.

Models come in all shapes and sizes. They are characterized qualitatively, often reflecting early-stage conceptualization of a problem, and then are defined quantitatively as more domain knowledge is acquired (Fishwick, 1995; Pidd, 2004; Robinson, 2004; Fishwick, 2007a). Let's consider an abstract model termed the Verhulst, or logistic, model. The function defining the model, and whose solution is a sigmoidal curve, was studied in the mid-19th century by François Verhulst, who was concerned with abstracting population growth dynamics:

This model seems like a good place to begin our discussion if we wanted to use this in a computer simulation involving systems biology (Murray, 2002) or ecology. But, there are some notable problems that may we encounter while using this model. The first is that while this equation is fairly canonical, there are other versions. We might avoid explicit use of the time variable t, add a rate variable, or introduce a carrying capacity. Each of these representations will refer to the same phrase ‘logistic equation’, while appearing slightly different in syntax. Furthermore, the equation has meaning within a physical context. The time-varying population P(t) may refer to gazelles or elephants where the equation is actually employed. The equation may appear in a scientific paper, a slide presentation, or in interactive mathematical software. The equation may be translated into one of a variety of computer programming languages. The meaning of the equation, if not immediately recognizable to students, must be found elsewhere in the same document where the equation is printed, or perhaps in another one. In summary, all aspects of the model, when taken as a whole, are completely disconnected.

What would be ideal is where we could interact with the equation, touching a variable or expression, and then being transported to code that implements the equation, or to a video of elephants showing a typical population whose dynamics are captured by the equation. Interacting with the differential operator might take us to a primer on differential equations. Not everyone who uses the equation would avail themselves of all of these human interactions, but many would. Where is the automatic linkage that reassures us that the equation is coded correctly when one must wade through separately located code to find the equation's translation? Are there conceptual (perhaps natural language or diagrammatic) models that bear a semantic connection to the equation, and is there an interactive mechanism to link between the two?

These problems run rampant through all models used in simulation. Model components have weak semantic, and correspondingly interactive, linkage to other components, media, and forms. This link paucity results in models that are difficult to understand, verify, and communicate with others. In a futuristic vision of modelling, all models and their components would be fully interactive. We could touch and maneuver variables and expressions to yield the same type of richness that we currently get from web pages and linked multimedia, and from the type of enhanced human interaction possible from game consoles and controllers. When models are static and non-interactive, we quite literally lose touch of their meaning and real-world relevance.

2. Hypermodelling

We define hypermodelling as the general theory and practice of linking system models and their components. This theory and practice subsumes and extends existing approaches that have been used within simulation to connect and integrate models (ref. Sections 4 and 5). While the concept of linkage among documents began with the formation of the index in book production within the middle ages, the idea of broadening this concept within a focus on human interaction was introduced by Nelson (1965) with the terms: hypermedia and hyperlinking. A hyperdocument is defined by features of the current World Wide Web, and can be traversed to interrelate information and media, extending previously static interpretations of documents. Hypermodelling represents a natural extension of this evolutionary web science trend in documentation to system modelling. In particular, we are concerned with how dynamic models and their components are interrelated within the human interface (Fishwick, 2004). Hypermodelling represents an attempt to combine previously successful research in system model composition and connectivity with new areas involving human–model interaction by using a variety of technologies, including mixed reality. The sorts of themes that should interest us in hypermodelling include the following activities at a formal, as well as human-interface, level:

  • Linking models together within one computer or among many

  • Linking system models to other types of models (eg, information, geometry)

  • Linking model components to media

  • Composing system models from heterogeneous model types

  • Creating custom representations of the same formal model

In the spirit of hypermedia, and hyperlinks, we must maintain a commitment to studying model connectivity both formally as well as through the human interface. It is sometimes a challenge to separate the two. For example, hyperlinking in two dimensions on simulated pages has become ubiquitous although is also possible within three dimensional virtual spaces and in the real world with the appropriate augmentation. In both of these cases, the underlying formal hypertext reference syntax and semantics are the same regardless of human interface.

3. Types of hypermodelling interactions

We posit different three sorts of connectivity among models and humans to be able to characterize modes. Figure 1 shows an attempt at this characterization.

Figure 1
figure 1

Three ways models are interconnected to support hypermodelling.

Full size image

Connecting models is not unlike connecting electrical or mechanical components; one must ensure an ‘impedance match’ using an electrical analogy or a matched information or network protocol on each side of the connection. The second type, integrative modelling, requires that models and their isomorphic properties be shown to be similar through special types of visualization and interaction. The difference between integrative modelling and the areas of composition and multimodelling lay in the addition of the human into the mix. If a human must see the connections between two models, then there must be explicit techniques employed to surface this connection. The third type, again involving humans, is where one type of model, taken abstractly, can be viewed and interacted-with differently depending on the target audience. Using the logistic equation as an example, the essence of the logistic equation may be shown in one of several equational forms, visually using feedback, or behaviourally using video and animation. In each customized form, the assumption is that the underlying model structure can be induced, but that the model presentation is different. We will spend less time on the first type since this has been studied at greater depth in the literature.

4. Composition and multimodelling

If we refer to models connecting with other models, these models are either of the same formal type or of different types. If they are of the same type, we refer to the model composition problem in simulation (Page and Opper, 1999; Davis and Anderson, 2004)—where we must concern ourselves with how models are coupled. If, however, the models are of different types then we enter the realm of heterogeneous models and components that must be married (Fishwick, 1991) to perform their aggregate functions. These two concepts are known by different names throughout the simulation literature. Composition may be referred to as coupling or interfacing, and multimodelling (Fishwick, 1995; Park et al, 2007) as multiformalism, multimodal, or multiscale since heterogeneity is common when we create a larger model composed of sub-models each capturing a different partition of space-time.

The connecting of model components is fairly straightforward within the same formalism since these formalisms contain intrinsic, recursively defined methods for extension. For example, in mathematical models, functional composition is the traditional means of expanding models to promote connectivity. Issues arise when the domain of the function differs: linking a continuous-time function with a discrete-time or discrete-event function requires careful set-theoretic definitions and algorithms (Zeigler et al, 2000). Often, the connectivity is not within a single formalism, but rather over a computer network. Standards such as Distributed Interactive Simulation (Neyland, 1997) and the subsequent High Level Architecture (Kuhl et al, 1999) cover approaches to connecting, not necessarily models, but most certainly computer programs whose function was to simulate individual nodes as part of a federation. The interest in grid computing (Taylor et al, 2010b) and workflows directly addresses model interconnectivity issues over a distance, and with scientific sharing, collaboration, and reuse in mind.

5. Integrative modelling

Figure 2 exemplifies a scenario (Ezzell et al, 2011) involving cardiovascular simulation.

Figure 2
figure 2

A cardiovascular simulation with two panes. The left pane shows a compartmental model emphasizing the physiology of the human heart, and the right pane shows an anatomically correct model of the heart co-located with the compartmental model.

Full size image

There are three models that are integrated into this interface: (1) compartmental, (2) anatomical, and (3) ontological. The compartmental model is a directed graph connecting ‘compartments’ that can accumulate and release flow based on resistances: this model is what most simulation researchers would term the ‘simulation model’ since its immediate underlying translation is a set of ordinary differential equations. The anatomical model is a spline-based mesh of the human heart, and when computer graphics researchers use the word ‘model’, the mesh would be one of their primary references for modelling. The ontological model defines a taxonomic set of relations capturing that a human has lungs, a heart, and so forth, and that the heart has certain sub-components. What is important about this interface, and not immediately apparent from a static image, is that all models and components have full interactive interconnections (Fishwick, 2004; Park and Fishwick, 2005). Clicking on the ‘right atrium’ (circled in the left pane) highlights the location of the atrium. This interaction is indicated with an arrow connected both panes. Also, this same interaction allows for the user to show a dynamic plot of pressure and compartment volume, which is not shown in this figure but more fully described in Ezzell et al (2011). A full-fledged interface would allow the user to see videos of the heart functioning, or surgical procedures, all through the same root interface. One is not required to rummage through an assortment of web pages or books to create the semantic hyperlinkage.

Figure 3 represents other work, as a collaboration with the University of Central Florida and the NASA Kennedy Space Center (Fishwick et al, 2007; Rabelo et al, 2011). The goal was to link a project management workflow directly to engineering objects that are referred to in that workflow. These objects are the space shuttle and the external tanks used to help launch the shuttle into orbit. Traditionally, project management views of the world are not integrated with animations or engineering designs, and so this experiment was to facilitate this interchange. One clicks on a project phase and all resources appear on the right in Figure 3. Inversely, clicking on an aspect of the design on the right-side of that figure highlights the corresponding project management tasks that are semantically connected.

Figure 3
figure 3

Integrative modelling, connecting a project management worksheet with a computer-aided design model of the shuttle-tank mating process, which occurs to prepare the space shuttle for launch.

Full size image

6. Model customization

People are very different in their requirements when it comes to the presentation of information. If someone is sitting down in the morning to read a newspaper, information is expected to be presented fairly quickly. That ‘reader’ has things to do and there is limited time for absorption of the information. However, if later, that same person goes into a museum, similar information can be presented, which may take longer to absorb, and there is less self-imposed time pressure. As computing technology progresses, it is often not necessary to shift physical locations to move from one context to another. The same tablet screen can be used to quickly scan line graphs or to enter virtual environments. Thus, the shift in how information is acquired becomes a mindset rather than a trip across town.

Since models, and their executions, are complex types of information, the value of the information on the audience can be considered across a wide range of human need. If quick comprehension is required to understand the process-based layout of an industrial plant, then a diagram is the appropriate language. If, however, attitudinal change, rhetorical power, or emotion plays a role, then non-diagrammatic forms may be useful. Figure 4 shows a typical layout for a System Dynamics flow graph for metabolism.

Figure 4
figure 4

A System Dynamics style flow graph capturing human metabolism Copyright Inderscience, Fishwick (2008).

Full size image

Figure 5 shows an isomorphic representation of Figure 4. The goals of Figures 4 and 5 are very different, and often the effects of each is not immediately known.

Figure 5
figure 5

An immersive representation of Figure 4, with an avatar, who can control the flow valves, shown on the left side Copyright Inderscience, Fishwick (2008).

Full size image

The way in which model components are shown, and interacted with, can be determined only through social and behavioural research methods. Recently, as part of two experiments (Fishwick et al, 2010), we introduced two different stimuli—one web-based and the other an immersive environment not unlike the one shown in Figure 5. While results from one experiment and specific context cannot be uniformly generalized to other contexts, our team found some interesting results in terms of memory and gender. For memory, in particular, the immersive environment resulted in statistically higher short- and long-term memory for all but the verbal information provided in the stimuli. Memory was tested for free recall and for recognition (ie, cued recall). This suggests that representations such as Figure 5 may be more than stunning pictures, and instead, use of these representations may result in cognitively different results and effects.

7. Concepts, technologies, and economic issues

The goals of enhancing the human's interaction with simulation models depends on what technologies and methods exist to support the interaction modes. We need basic concepts to support the interactions and new technologies to implement these concepts. There will always be issues to accompany technology applications. Let's consider integrative modelling first and then proceed to model customization.

To make a solid integration, we need concepts at two levels: a way to formalize a connection, and a way to manifest this formal connection into the human-model interface. There are many ways to formalize relations, but perhaps the most complete are the standards and research surrounding the semantic web (Berners-Lee et al, 2001). A hyperlink can be formally defined independent of its interaction implementation. The formal mechanisms relating to ontology formation are central to classifying and connecting model information (Fishwick and Miller, 2004; Silver et al, 2006; Bell et al, 2008; Tolk and Turnitsa, 2007; Taylor et al, 2010a). To integrate models together within a human-model implementation, we can rely on two methods: co-location and co-interaction. We define co-location to be where two models and their components are juxtaposed within an interface through proximity, connectivity, or encapsulation. Placing one model next to another is an example of proximity-based integration—not unlike the strategy of associating a figure caption with the figure: the two are co-located by being next to each other. In other situations, a correspondence can be obtained through drawing arcs or arrows (eg, connectivity) or by placing one inside the other (eg, encapsulation). In the diagram on the right-side of Figure 2, there are two models: the mesh and the compartmental model whose topological structure is twisted to conform to the mesh geometry. These correspondence approaches are discussed in Fishwick (2007b). We define co-interaction through the capability of integrating by having a human interaction in one model be reflected by a corresponding change in the second model. Figures 2 and 3 are examples of this, where manipulation of one model on the left results in an immediate set of visual cues on the right-most model, and vice-versa. Technologies to support integrative modelling are numerous, from the web-browser mouse-based hyperlink page transitions to the use of mixed reality to co-locate models with reality (Quarles et al, 2009). Perhaps the easiest way to think of co-interaction, generally, is with media controllers: gestures on a hand-held controller or tablet immediately result in changes in the media presentation.

In the area of model customization, concepts that are employed are based on the type of model information to convey. Typically, more abstract levels are encoded in mathematical text. Less abstract levels may use diagrams, and interfaces that rely on more advanced technologies may use shading, colour, textures, depth, additional space dimensions, and stereoscopy. The philosophical area of aesthetics can be defined by enumerating the manifold ways in which humans interact with artifacts, including purely cognitive principles (eg, minimalism) as well as principles found in design and the arts where alternative representations serve multiple constituencies, goals, and cognitive effects. Aesthetic computing (Fishwick, 2006, 2008) is an area where techniques and concepts in the field of aesthetics influence information and model representations.

Creating integrations and model customizations can be expensive, and may require new types of tools. Returning to the logistic equation that we covered at the start of the article, there is a dearth of tools that will allow users to interact with variables and terms to obtain corresponding models, underlying source code implementation, or media. Existing tools may not be advanced enough to support this kind of interactivity. Creating an interface with two models (compartmental and anatomical, as in Figure 2) is certainly more time-consuming than a single model interface. Model customization suffers from the same problem: creating one version of a model is less expensive than creating multiple versions, each for a different target audience or demographic. The immersive model in Figure 5 took much longer to create than the diagram in Figure 4. However, the methods behind model integration and customization are certainly not the only factors dictating relatively high modelling costs. Models that are complex and have many components and interactions pose economic challenges, and more mathematical approaches based on systems theory have issues similar to those of formal methods in software analysis: the models may be difficult for a majority to use on a regular basis. In the case of formal methods, they are often relegated to safety and security (ie, high risk) computational situations because of this reason.

Despite these economic problems, the advantages of integrative modelling and model customization are significant enough that we need to research faster and cheaper approaches. It is possible that increased reuse of models, diagrams, and component representations in a supply chain fashion may reduce the costs of model integration. It is also assumed that improved integrative modelling in the human interface will aid in adoption of all levels of model abstraction—from systems theoretic to highly conceptual, since the inter-linkages facilitate learning and comprehension of the models in the same way that hyperlinks improve understanding of a subject through the linking of multiple documents.

8. Summary and future work

Even though the World Wide Web has better reconnected us to information, we have a long way to go in connecting the system models that we use to each other, and to reality. These connections must take into account formal composability, and interactivity, but also the human factor. We must investigate better approaches along two fronts, at least: (1) inter-relating different models, and their semantics, within the human-computer interface, and (2) presenting the same underlying formal model in many different ways, and with different interaction modalities, depending on target audience and the goal. Single model formalisms, and interactions, are ultimately insufficient since they suffer the problem of isolation. We’ve overviewed some techniques for achieving this goal, and also discussed the economic impediments—that to have multiple models, whether to represent different abstractions or for different demographics, we need better tools for mathematics, models, and software engineering. Being able to click on an equational term in a slide and be presented with the appropriate code segment implementing that term, or with media that explains the term, is also going to take a new mindset beyond advances in tool development. As with web page content and with real-world objects, we have to think of simulation models as fully interactive structures rather than as static entities.

What will need to happen to make it so that integrative and custom modelling will be made more economical and readily available? The semantic web provides a formal structure for semantic model component linkage, and yet, someone or something has to create the links. One possibility is that links will begin with the manufacturer of physical items, where these items will ship with, not only instruction manuals, but also completely linked model information. Reduction in technology costs, as with smartphones, tablets, and mixed/augmented reality equipment, will also help since these technologies assist in constructing integrative modelling interfaces. For model customization, costs are reduced through diagram and mesh reuse and the formation of a marketplace for visual model components. Automation, in general, has to increase since without it, modellers are forced to hand craft each model.