1 Introduction

The increasing threat to military forces of improvised explosives devices (IED) (Winter et al. 2008; US Marine Corps 2006) has led to considerable attention by the Defence analyst community to identify interventions such as adoptions of science and technology options or changes to procedure or organisational structure. In addition to providing insights to the problem at hand, it has also allowed us to explore the role of modelling in developing a sound Operations Research (OR) framework, in particular in addressing the earlier phases of the study.

For an IED system, a simple description involves defining two sides: blue (friendly) and red (enemy) that conduct concurrent actions with competing aims and objectives. Our initial intuition was to treat the counter-IED as a ‘problem’ rather than a ‘mess’ (Pidd 2004a) as there is already partial appreciation of the factors involved as it is similar in concept to other military actions. The corollary of this, of course, is that for any ‘problem’ there will not necessarily be a single ‘solution’—see Chapter 2 in (NATO RTO SAS-087 2012a). Thus, we were confident that the first step should be to devise a detailed model and use this to understand and explore the IED system and so provide an audit trail to support any proposed changes. A system-level diagram as shown in Fig. 1 is an approach that links specific blue and red force actions within overall campaign objectives. Any model(s) would reflect on the interfaces between blue force actions and those conventional and specific IED red force actions. Military forces using, or operating under threat of, IEDs can be viewed as a human activity system pursuing specific goals. Such an approach is then amenable to exploration in one form or another. We were, however, readily able to identify two methods that would provide useful models of the concepts contained in Fig. 1, in particular the interfaces between blue and red forces.

Fig. 1
figure 1

A system diagram for a blue force in conflict with a red force that is using IEDs (items in italics are those of primary interest)—grey areas show the interfaces between the two sides

Full size image

That more than one method could be readily identified for such a study led us to consider whether it might be useful to generate an underlying conceptual rationale to guide selection of an appropriate model to base the study on. We use the term ‘substrate’ for such a model to indicate the capture of common, underlying principles of analysis and synthesis that would guide both methods. A substrate model can thus be viewed as a mechanism for formally explicating and bringing together the various aspects that make up the context of the study. In effect it becomes a constant that is shared by the analysts and problem owner and provides an agreed reference that supports an audit trail to support material presented to decision makers. The notion of a fixed element is entirely consistent with soft OR approaches (see later) and, importantly, provides a generalised basis for guiding a study irrespective of the specific modelling method used. Particularly, we feel it can be used to guide a study in a way that can ameliorate the potential to introduce biases or focus on less relevant aspects of the problem space. Using the concept as outlined, the position of a substrate model for tackling the IED problem space outline above is shown in Fig. 2.

Fig. 2
figure 2

The complementary role of the ‘substrate model’ associated with the problem-structuring study model

Full size image

Four major features are implicit in Fig. 2: the fixed element of the substrate model, its source material and agreed issues of concern (the large oval); the study actors (analysts and problem owners); the soft OR process (shaded area); and finally the study products (in this a case a set of options for further consideration).

Modelling is key to the practice of such ‘soft’ OR (Franco and Montibeller 2010) and good practice (Smits 2010) suggests such a model can be used to clarify thinking and structure an investigation around processes, policy decisions (i.e. intervention points) and performance indicators. The substrate model is derived from perception of the environment strengthened through objective observation of actual experience and plays a focussing role in the study. Central to the development of the substrate model is an agreed description of the system under study which could be used in other analyses. Such an approach has been referred to as the use of models as ‘transitional objects’ (Franco and Rouwette 2011). In a recent paper (Franco 2013), the concept of models as boundary objects was proposed and inter alia the notions of tangibility and associability mentioned as contributing to useability in line with our opinions on the nature of such models. The initial results of the study, the ‘issues of concern’, should be seen as completing the agreed and shared understanding of the analysis.

These issues of concern arise though application of the methodologies implicit to ‘problem structuring’ (Mingers 2011; Ackermann 2012; Pidd 2003, 2004b; Reynolds and Holwell 2010; White 2009) and the shaded area in Fig. 2 can be seen as encapsulating this process. Irrespective of the techniques used, the common theme is that problem owners and the analysts need to work together to achieve an outcome where much of the input material is uncertain or at least based on judgement (NATO RTO SAS-087 2012a). The combination of the soft OR process with the use of a substrate model gives a conceptual ‘front-end’ understanding, concentrating on initial appraisal of the scope of the problem space and leading to better-considered design of the analytic study. It is thus built on ‘soft system thinking’ (Checkland and Holwell 2004). The discovery of issues is integral to soft systems approaches and the development of models tailored for system investigation and improvement, or providing insights for debate (Pidd 2010). A danger in such identification of issues is the potential to focus on the wrong aspects of the problem space when options (the product of the study) are presented. This can lead to criticisms of bias or the study not being relevant.

In this paper, we describe the use of two modelling methods to analyse the counter-IED problem. We show how two distinct approaches may be taken, the type of the results that may be expected and how the notion of the substrate model is consistent with the early stages of an analysis (problem structuring) and can mitigate any potential problems of bias or focus. We also provide a comparison of the results obtained by the two models and comment on the use of both as substrate models.

2 Modelling strategy

Four major features were considered in developing our modelling strategy. First, we reviewed the two modelling approaches, second we investigated how the models could be brought together within a sound framework to structure the analysis, third we examined the available information sources and finally, we considered the potential interests and goals of the client.

2.1 Modelling approaches

In previous work with similar ‘front-end’ studies we have successfully used two distinct modelling approaches as follows. In the first, a sequential description of the stages involved in a military action is expressed as a series of events, focussing on ‘what is done’ and in what order. This mimics the individual events as they occur. For the second approach, we applied a functional description, focussing on ‘how it is done’ expressed in terms of an influence diagram (the term is used here in the broadest sense), relating elements contributing to a higher goal. In effect, this mimics how features of the overall system evolve over time or when exposed to an external stimulus. Respectively, our interpretation is that these represent distinct descriptions with different model complexity (linear versus branched), level of detail (intimate versus holistic), tangibility (concrete versus conceptual), statistical treatment (stochastic versus deterministic), analytical viewpoint (systematic versus systemic), type of question (what is being done? versus how is it being done?) and likely advice to the decision makers (specific versus general interventions).

In its simplest form, the IED problem can be treated as a two-sided system. Friendly forces (‘blue’) compete with opposing forces (‘red’) though for modern operations we note the presence of non-combatants such as civilians (‘white’) and aid-agencies (‘green’). Often blue and red have different objectives and red may attempt to gain asymmetric effects through targeting civilians. This then presents a dilemma to blue—should an operation be focussed on achieving blue’s outcomes in an IED environment (such as peacekeeping) or preventing red forces developing or employing IEDs? The first of these paradigms is reactive, focussed on defensive measures directed towards carrying out operations in an IED environment, while countering anticipated threats. In contrast, the second is proactive, leading to an offensive stance specifically targeting all aspects of red’s use of IEDs through identifying blue opportunities (Winter et al. 2008).

For the purposes of this study, we chose to limit the scope of the model to description of military actions conducted in an IED environment. While we have not addressed the higher level issues of modelling the wider issues, such as identifying options to prevent IEDs being used in the first place (Eisler 2012), there is no reason why the substrate models cannot be used for this purpose.

Finally, while we do not claim that these are the only two modelling methods that could be suitable, they are quite distinct and both provide an empirical means of deriving a model. Importantly, practical application to the counter-IED example allows comment to be made on their applicability as substrate models.

2.2 Analytical framework

Useful compilations have been made of those established techniques inherent to PSMs that could be exploited to gather data and lead to development of a shared substrate model (Reynolds and Holwell 2010; Pidd 2004b). A survey of recent literature (Howick and Ackermann 2011) suggests that Soft Systems Methodology, Strategic Choice Analysis and Strategic Options Decision Analysis/Journey Making dominate the published literature and all of these contain techniques that could be applied in this case. Successful implementation of mixed methods was also noted, including qualitative and quantitative techniques. Informal approaches such as seminar wargames, often referred to (in Defence) as Bunch of Guys Sitting Around a Table [BOGSATs (Rubel 2006)] may also be used. Whatever approaches are used to develop representation of the system under study, our experience is that it is sound to introduce some possible ways for the analysts or workshop participants to ‘start the ball rolling’.

There are several ways to describe military actions and the system in general (Bowley et al. 1998) and these can be used to provide structure for the models. Two convenient methods that we have developed are the notions of ‘activities’ and ‘core skills’. The term ‘activities’ describe actions that are common to dismounted combatant operations (Curtis 2000). Examples are ambush, patrol and vehicle check point. They are discrete and can be the subject of a wargame or simulation with semi-quantitative (but subjective) higher level measures proposed. An example of a measure of outcome (Curtis and Bowley 1999) would be that an assault was considered a success if the position was taken within 20 min, casualties were less than 10 %, and the unit is equipped to withstand an immediate counterattack. For this description, the constituent elements can be considered as lower level measures of effectiveness. Importantly, they represent a mid-point entry into the analysis and each can be readily broken down into stages or can be amalgamated into larger groupings (Hobbs and Curtis 2011).

‘Core skills’ take a similar comprehensive approach, but in this case the enduring capabilities required for Army operations are considered (Curtis and Dortmans 2004). Unlike the open-ended list of activities this set is limited to seven: engagement, information collection, protection, movement, communication, sustainment and decision making. These are harder to quantify, relying upon subjective combinations of representative constituents in the same way that multicriteria decision making is addressed, and indeed may be used for that purpose—to identify these factors (see later). Core skills are also valuable in grouping like capabilities such as equipment used across several activities, and thus providing prioritisation insights based on frequency. In passing, it might be noted that in the IED literature a similar approach is used through grouping based on the operational domains of predict, prevent, detect, neutralize, protect and avoid (US Marine Corps 2006).

2.3 Information availability

For development of substrate models two sources of information are immediately apparent: interviews of subject matter experts or informed analyst interpretation of existing material. In the first, the (blue) subject matter experts (SMEs) are asked ‘how they do it’ or possibly ‘how they would like to do it’ which gives a focus on stakeholder realism. While a typical approach is to have a facilitated workshop of SMEs (Franco and Montibeller 2010; Franco and Rouwette 2011), a mixed strategy can be adopted where a distributed interaction allows the analyst to pre-prepare a model (Morton et al. 2007). This could then be taken (Lane and Huseman 2008) to a workshop for discussion. While this may lead to consensus, in the present case the red force aspects (red teaming (Ministry of Defence (UK) 2010) would be speculative and any model suffers from a lack of confirmatory observations by the analyst such as through field observations (Rees and Curtis 2013). There is thus only incomplete contribution by all the system stakeholders. This might be compared to a study of health care where, by and large, both the medical provider and patient aspects are either well documented or can be monitored. This observation, and the preceding discussion exemplify why the extensive literature on modelling, for example, the health system, including discrete-event simulation (DES) leading to systems dynamics-based optimisation (Vanderby and Carter 2010; Smits 2010; Davies et al. 2000; Fone et al. 2003; Robinson 2005) is not applicable to this case.

The second approach is for experienced analysts to use documentation (such as manuals), observations (Hobbs and Curtis 2011) and comparison with other Army activities to derive the models, thus giving a focus on rigour and independence. Ab initio studies such as these may also lead to new insights. For credibility, such models would then need to be approved by the SME group.

2.4 Client interests

Military capabilities can be described in terms of a Defence capability development chain which can be related to the concept of a System Life-Cycle (Moon et al. 2013). This can also be used to identify relevant clients and stakeholders for the study undertaken. Table 1 illustrates the phases of the Defence capability development chain as applied in Australia (Department of Defence (Australia) 2006) and then links them to the international standard for the system life cycle (Software and Systems Engineering Standards Committee of the IEEE Computer Society 2008). The types of OR studies undertaken are highly dependent on the needs of the clients, ranging from very initial insights to scope a military capability, through identification of specific options to detailed analyses of prioritised acquisitions.

Table 1 Relevant OR aspects of the phases of the Defence capability development chain
Full size table

The threat of IEDs is both recurrent and immediate, so resolution of issues may involve a mix of strategies aimed at long- and medium-term responses. In the first, a pure needs phase approach would be appropriate to adopt a longer-term approach, while in the second the overall acquisition process may be compressed and include the requirements phase to frame technical specifications. Our strategy was thus to maintain flexibility and ensure that both needs and requirements aspects could be addressed. These both lie in the general ‘front-end C3’ area of Contexts, Concepts and Capability options with an emphasis on providing insights into possibilities for debate rather a detailed plan of action. Thus, it is important to ensure that the goals of validity, credibility and acceptance (NATO RTO SAS-087 2012b) are met through a sound process, including a documented audit trail.

2.5 Approach taken

As we wished to also explore the broader application of the modelling methods used, we decided on an inclusive approach covering several of the possible combinations implied above. For practical reasons Defence clients were not actively involved in the study. The sourcing and preparation of the information used and application of the models was undertaken by the analysts using their experience and drawing upon their extensive expertise and analytical understanding of military operations.

3 The example—Vehicle check point (VCP)

A VCP was chosen as the case study for illustrating the comparative approach outlined here. This is a good example in that it is easy to visualise and understand for a general reader who may not be familiar with military operations. Importantly, it can be described in broad terms without need for introducing a detailed discussion of military tactics, techniques and procedures.

A VCP can be thought of as a road block constructed to control access to a facility, town or region. As noted by (Curtis and Hobbs 1997), vehicles approaching the VCP are assessed at a pre-determined distance, channelled through some sort of traffic control, inspected and allowed to pass only if deemed safe to do so. The threat of IEDs presents an added complication to the operation of a VCP as illustrated by the following example of a higher level measure of outcome (Curtis and Bowley 1999). ‘The VCP activity was successful if 95 % of suspect vehicles were identified before they approached the site, IEDs were detected at a safe range and any offensive use of IEDs caused minimal damage, while still allowing legitimate traffic to cross.’

Using the VCP as an example, issues can be identified using both methods guided by a substrate model. The means of addressing such issues may be then be grouped into three categories: science and technology interventions (new equipment or new use of existing equipment), changed procedures or instituting alternative organisational structures. Costing, training or other implications of options for addressing the issues can then be compared. For instance, domestic security concern over the availability of ammonium nitrate might lead to a technology response (e.g. a nitrogen quadrupole detector) or to place restrictions on access to the raw material. The indicative material presented in Tables 2 and 3 thus does not provide a preferred means of dealing with the issues, but rather acts as an audit trail to identify an extensive set of possible options as to a course of action to be taken. However, for the illustrative purposes of this paper we will describe the results only in terms of science and technology options.

Table 2 Example sequential analysis of part of a VCP activity
Full size table
Table 3 Example influence diagram derived issues for a VCP activity
Full size table

4 Modelling based on a series of events

Sequential methods can be based on an Integrated computer aided manufacturing DEFinition for functional modelling (IDEF0) approach, whereby each transition is governed by ‘rules’ and ‘tools’ (Systems Management College 2001). Other formalisms (Curtis et al. 2006) include SSM, Stella/iTHINK and systemigrams. There is considerable literature on the use of DES models to provide insights into processes (e.g. see Robinson 2008; Jun et al. 1999; Robinson 2005; Duguay and Chetouane 2007; Gunnal and Pidd 2010) though they tend to be quantitative in nature. In our usage, we use DES as a qualitative tool to portray the process to aid in the qualitative analysis.

Sequential methods have been used to describe military actions as a substrate to support seminar wargames such as for identifying science and technology implications for the next generation of soldier modernisation (Curtis 2000). The essence is that each box represents a discrete action that is performed while the arrows leading in and out link events.

In the current context, it is then straightforward to determine a higher level sequence of events that could occur in an expeditionary operation (Fig. 3). The rectangles represent activities; each could be described in greater detail provided there is not substantial repetition of specific events in the sequence of activities identified. Taken together, the activities represent an overall perspective for the analyst.

Fig. 3
figure 3

Simplified sequence of events centred at an expeditionary base

Full size image

For the current study, we considered two aspects in constructing the sequential diagrams for the individual activities (Fig. 4): the blue forces carrying out their campaign while protecting against potential actions implicit in the red threat environment, and secondly taking a proactive stance against the red force operation (the dilemma presented before). The two cases are similar in that there is an interface at an intimate level that can be described in both narrative and semi-quantitative terms, and potential options identified (Fig. 5). Issues arise at the interface.

Fig. 4
figure 4

Blue and red aspects of the sequential analysis with blue’s reactive stance shown on the left and a proactive approach on the right

Full size image
Fig. 5
figure 5

Interface (grey area) between blue and red actions in a sequential analysis with blue’s reactive stance (upper) and proactive approach (lower)

Full size image

Typically these models are in the form of a sequence of events that are linked to the next stage (rarely to more than one) and can be conveniently portrayed in diagrammatic or tabular form. Table 2 gives an example of how issues can then be generated for the case of a vehicle approaching a check point. Possible options to deal with the situation can then be linked to these issues, with an example being the use of non-lethal weapons to disable a vehicle that did not answer requests to stop. This should be viewed as part of a larger table for a VCP and has been constructed in terms of systematically working through the individual stages, considering what needs to be done, identifying possible vulnerabilities and proposing interventions. Some of these science and technology interventions would include use of existing technologies, others would require using existing technologies in novel ways and some might require the development of new technology. This latter observation may trigger a prioritised research programme.

Our experiences of using this approach are as follows. Describing the activity as a set of linked events is relatively straightforward, but does require sufficient contextual knowledge of military tactics, techniques and procedures by the analyst. We would have confidence that similar (but not exactly the same) diagrams would be produced but analysts with equivalent levels of experience. The templates themselves are readily understood by SMEs allowing verification and if necessary refinement and modification.

Considerable value-adding occurs when each step is subject to a protocol of interrogation (Curtis 2000; Curtis et al. 2006) and shows the value of having a substrate model. For instance, the practice of walking down a street as part of a patrol activity can be explored in a workshop in terms of the information collection core skill and this may lead to an issue of the inability to know what is on the other side of a wall. Potential solutions like enhanced language and communication skills to talk to the inhabitants, or uptake of a technology solution can be generated. The process can then be repeated with another core skill until all aspects of that particular step have been explored. With experience, the analyst can identify parallel streams or variations thus avoiding duplication and reducing the complexity of the model. For instance, weapons usage may involve blast, fragmentation or kinetic means as well as several types of warhead.

While the process is straightforward, the model can be extremely complex. For instance, the full VCP example had 8 discrete stages (only 3 are shown in Table 2), and once the prompts of the seven core skills and variations are taken into account, over 50 requirements or issues were identified. When other activities were added to list, such as observation post and crowd control, hundreds of issues were identified, though of course many were repeated.

5 Modelling based on functional (‘influence’) diagrams

At their simplest, influence diagrams contain the features of variables, linked flows with polarities and feedback loops though they are often more complex and include quantifiable levels, flow rates and delays—see Chapter 2 in (Coyle 2004). We have adapted some of these notions to represent the system, but recognise our usage is subtly different. We start from an overall goal and use this to derive constituent components (Fig. 6)—see later. Thus, position in the diagram infers meaning on the profile of a particular variable (Curtis and Dortmans 2004). As a substrate model, the purpose is to show linkages between features of interest in particular how changes (perturbations) may lead to more or less effective outcomes in a simple manner. Thus ‘what if’ speculations may occur with a common audit trail to rationalise insights. This will contrast with cases where features are quantifiable and which could lead to a system dynamics approach, such as for military (Coyle et al. 1999; Artelli et al. 2009; Homer and Hirsch 2006) and health (Smits 2010) studies. Perhaps the key difference in this work is that we note that our variables are unquantifiable and for this reason literature comparing DES with systems dynamics while useful for comparing model development is of less use in discussion about outputs (Tako and Robinson 2010, 2009; Brailsford and Hilton 2001).

Fig. 6
figure 6

Simplified ‘influence diagram’ for a VCP (Letters A–E refer to Table 2)

Full size image

The system-level diagram (Fig. 1) shows the rationale behind this approach. The essence of the model is that blue has an overarching campaign objective that is carried out by specific actions within this context. Red forces will be driven by a higher objective that could include ‘conventional’ (i.e. not including IEDs) and ‘non-conventional’ (IEDs) aspects. The latter will be directed towards countering specific blue force actions, disrupting the overall blue campaign as well as contributing to the red force goals. For the purposes of this study, it was felt that prospective clients needed insights to improve those italicised areas of the diagram represented by the two action pairs:

  1. 1.

    Blue action to counter red action//‘conventional’ red actions to stop or hinder blue and.

  2. 2.

    Specific blue action to counter IEDs//direct red IED actions.

Pair 1 is complicated by the fact that blue has to assume that IEDs could be used, whether they were or not, and that blue should operate in an environment of uncertainty and prepare accordingly. These approaches are similar to the respective reactive and proactive stances described in Fig. 4 although with the important difference that red force’s actions would be treated in a continuous manner in both cases.

Two options exist for the next stage—identification of ‘blue force actions’ (Fig. 1) to be analysed—either by core skills or activities. Both are compatible with the notion of influences though in our experience of the use of influence diagrams to describe core skills (Curtis and Dortmans 2004), we felt that this was too abstract for the current purposes of looking at a specific action like the use or threat of IEDs. For this reason, we chose to use ‘activities’ for the principal means of analysis. This also gave us the ability to compare the results with the sequential analysis approach.

In our current usage, position in the diagram has meaning (Fig. 6) and this may be contrasted with those studies where there are multiple features that need to be balanced with none necessarily being more important than the others. The approach of identifying a higher level goal (a top-down approach) and building diagrams to explore the constituent features is common to problem-structuring methods, though occurs in several forms. A recent approach was the use of a prerequisites tree assists in identifying obstacles for achieving the goal (Clark and Mabin 2011). Other examples seem to be more bottom-up derived through examination of constituent issues. For instance, Soft Systems Modelling and values-focussed thinking have been used to develop fundamental organisational objectives in a manner that while expressed as a hierarchy (Neves et al. 2009) could have been portrayed as an influence diagram. A causal map was also used to develop a value tree (Belton and Stewart 2010). While the cognitive mapping process is subtly different in that it deals with individual beliefs (Franco and Lord 2011) rather than material processes, it still relies upon identification of key features within a framework.

Construction of such an ‘influence diagram’ is deceptively simple. Whereas in the sequential method it is a question of working one’s way through the events in a stepwise manner, in this case it is best to start from consideration of a higher level goal (called a measure of outcome—see before). The process can be considered like this—‘to achieve this goal what needs to be done?’ This is a creative process and is similar to the notion of multicriteria decision making (Neves et al. 2009) or the way that a measure of outcome is composed of lower level measures of effectiveness (Curtis and Bowley 1999). This might be compared to the (bottom-up) case where all the likely factors are listed and then linked up to form an influence diagram.

Core skills are useful in this regard to focus thinking on the major constituents. This is best explained by example, as follows based on the description of the VCP (Fig. 6). As shown there are three primary inputs (in ovals with solid lines) to this high-level measure: ability to identify red, capability to stop vehicles and the ability to defend if an IED detonates at the check point. These correspond to the core skills of information collection, engagement and protection, respectively. All contribute positively in standard influence diagram nomenclature. The rest of the diagram has been derived in a similar manner. Those items placed in rectangles represent peripheral features or those that directly contribute to these primary inputs and are principally included for clarity. The ovals with the dashed and dotted lines represent further nuances to the system that needs to be explored separately. Based on the position within the diagram these are described as secondary and tertiary inputs, respectively. While this approach may lead to multiple incidences of a particular issue, this is preferable to missing one entirely. This distinction also allows a lower weighting to be placed on any issues found there. Although not discussed further, Fig. 6 includes a loop to capture how red could respond. As shown here, blue success at the VCP would lead to a change in red behaviour through learning of blue capability. The diagram could thus have value in driving a campaign analysis including red team responses.

Table 3 shows issues generated by this method. As far as possible, consistent terminology has been used to that for Table 2. As for the sequential case all the influences (shown as labelled arrows on Fig. 6) can be examined by a series of prompts and can be grouped as before. In contrast to the previous case, specific deficiencies cannot be identified.

Despite the intricacy of the diagrams, the influence diagram led to about 15 issues being identified for the VCP example, considerably less than for the sequential case (over 50). We argue that this is because of lower duplication or the grouping of more detailed items rather than a failure in the method to cover the range of issues adequately. To highlight this, parallel ways to achieve the same ends are not considered, just that the outcome needs to be achieved.

6 Comparison of the results from the two methods

Given the empirical nature of this research, it is necessary to adopt an a posteriori interpretation of the results derived by the two methods as the means of comparison. Thus, we examine the three features of the identified issues, their frequency and their pattern.

For the sequential approach, we identified 161 discrete science and technology options (as potential solutions to issues of concern) to consider once all the activities were included. Many of these were found across the spread of activities and in total were represented up to 40 times each. Other examples were far less widespread and about half had one or two occurrences. Identification of the core skill most relevant to each of these 161 options allowed consolidation into 18 larger issue groupings (Fig. 7) and each could be further assigned to one of six core skills (the movement skill was not included in this study). Thus, two key insights were obtained from this analysis: the number of occurrences and the spread across the activities. These can be used as an indication of importance of the issues although such insights should be treated with caution and only used for broad prioritisation.

Fig. 7
figure 7

Comparison of the distribution of issues by the two methods

Full size image

This grouping system proved compatible with the functional method and again the issues could be readily combined into the 18 larger groupings (Fig. 7). We found no substantially new issues through using the second method. However, the two studies were conducted at about the same time with the same analysts so we are reluctant to comment if one method is likely to discover more issues than the other. We did find, however, that the total number of occurrences was lower (340 opposed to 740) and that the distribution was different. The latter observation is a cause for comment if not concern.

This difference in emphasis is more apparent (Fig. 8) when the results were analysed by the six core skills used it this study. These results are expressed in relative terms, taking into account the differing number of occurrences between the two methods. More concrete aspects, including so called ‘kinetic’ effects (e.g. protection, information collection and engagement) were dominant for the sequential case, but for the functional method there was more recognition of less tangible aspects with for instance decision making rising in prominence. This is probably due to two factors. First, it is easier to describe ‘kinetic’ events in the sequential model, possibly in more detail than necessary, thus providing the potential for overrepresentation of the issues. Second, the analysts may also find it easier to consider such concrete actions in their perceptions of the events and to include specific items. This may arise because a readily identifiable piece of equipment is used or something can be readily measured as being capable or not. To some extent, the sequential analysis promotes reaction by the analyst and thus may favour more objective aspects. In contrast the functional approach is more proactive and triggers a systematic approach to answering the question of ‘how is this done?’. In addition, the subjective features may be inferred through training or past experiences and are thus not readily amenable to quantitative evaluation.

Fig. 8
figure 8

Distribution of issues by core skill for the two methods

Full size image

7 Discussion

We consider three aspects in our appreciation of this study. First, we examine if the notion of the substrate model is valid for this sort of analysis. Next, we explore the applicability of the two modelling methods in the role of substrate model and finally, we comment on the differences that were found in the results and how this might provide insights for future studies.

We found that the notion of a substrate model was a valuable part of the overall analytical process. Not only did the models provide a common reference point, but they also provided an audit trail to identify the necessary levels of transparency and credibility. The use of a substrate model as distinct from the soft OR process provides another means of seeking endorsement by the problem owner as well as identifying the sort of resources or access to expertise that may be required in populating the models (Rees and Curtis 2013). Identification of waypoints in an analysis is also a key part of a Code of Best Practice for such judgement-based studies (Wijnmalen and Curtis 2013). If the process outlined in Fig. 2 is adhered to then there is separation of an agreed and auditable model from the visible and invisible aspects of problem structuring—see Chapter 8 in (NATO RTO SAS-087 2012a). It also serves to differentiate the model from the people involved in the process and thus can be seen as an aid for progressing the study which can only be achieved by involvement of the actors (analysts, problem owners and other stakeholders) themselves (Hämäläinen et al. 2013). Validity of a model is critical to the acceptance of any results even if the final stages include the use of a ‘black box’ simulation, such as the process from the cause mapping to system dynamic modelling (Howick et al. 2008). The substrate models can be contributed to by the problems owners or other stakeholders and agreed as representing the system. Finally, we note that the substrate models have re-use capability once documented.

The two methods themselves also appear to be compatible with the established precedence of problem-structuring/soft OR methodologies. Many of the problem-structuring approaches presuppose a model or that the issues have already been identified though some have implicit tools embedded in the methodology. For instance, both the ‘how’ and ‘why’ are implicit in formulation of rich pictures (Checkland and Scholes 1990) and can be used as a source of issues (Hindle and Franco 2009). Inspection of the recent literature shows that soft OR case studies usually involve some sort of model, such as through rich pictures (Bell and Morse 2013), scenario analysis (Cole and Kuhlmann 2012), cognitive mapping (Franco and Lord 2011), viable system modelling (Espinosa and Walker 2013), causal loop diagrams (Smits 2010), and tree constructs for multicriteria decision analysis (Barford 2012).

The selection of an actual method or theme for many studies may depend on the final form of a study (Tako and Robinson 2010; Cooper et al. 2007; Sweetser 1999). For instance, if a system dynamics approach looking at the strategic or conceptual level will be used then an influence diagram will be a precursor. On the other hand, sequential analysis will lead to optimisation studies probably pitched at the tactical decisions. Of course, recent problem-structuring literature contains several references to, and examples of, ‘mixed methods’ or ‘multimethodology’ (Hindle and Franco 2009) and reasons are given for why one methodology may be picked over another, which may include familiarity (Howick and Ackermann 2011). This implies that there is likely to be no set way of conducting such analysis and that given time and resources more than one method may be tried.

Literature precedence is, however, sparse for the current example where we are looking for qualitative identification of issues. We thus present a comparison based on four overall headings: analyst useability, validity, client perception, and finally fitness for purpose (Table 4). This is a modification of a previous classification system (Tako and Robinson 2009) though we believe that the current sectors better reflect the nature of the soft OR process as described in Fig. 2.

Table 4 Comparison of the sequential and functional approaches
Full size table

7.1 Analyst usability

Both methods provide plausible analytic frameworks for both the higher (Figs. 1, 3) and detailed (Figs. 4, 5, 6) levels of analysis. Such models of blue activity can be readily developed by the analysts and further strengthened by the participation of the SMEs. The methods are both systematic, but are different in practice and can require different mind sets to achieve them (Franco and Meadows 2007; Hobbs and Curtis 2011).

The sequential analysis diagram is developed by a stepwise process of breaking an activity down into smaller actions starting with planning and ending with a completed action and recovery process. Typically, it is a linear process with little ambiguity about the position of a discrete action within the overall model. The level of detail is a matter for judgement and probably depends on the time available and expertise of the participants. The model is, however, likely to comprise an extensive list of actions and, while this is in a familiar form for testing by the SME, it is a lengthy process. Completeness and inclusiveness can be achieved through identification of parallel activities. The same process when applied to the red activity is more difficult to check and may suffer from a blue bias in construction. Application of the model to identify issues of concern is relatively straightforward as to some extent each item is treated in a methodical manner, such as with the use of a standard set of prompts.

In contrast, the functional diagram requires consideration of the essence of the activity to identify the constituents in an influence diagram-like format. The notion of defining a high-level goal for each activity as the first step, however, leads to a ready means of developing the model. There will still, however, be ambiguity in placing of elements and indeed the same action may be represented in more than one place in the diagram. It is a creative exercise to develop the basic notions of influences and the need for a loop to allow interaction of blue and red. While the two-dimensional nature of the influence diagram may seem more complicated, the fact that all the variations or alternatives that would be listed for a sequential model are subsumed into the general nature of the blue–red interaction means that it is more versatile in terms of update or when thinking about new ways of doing things, or updating current understanding. Application of the model to discover issues of concern is more difficult than for the sequential case, as it is more open ended but this raises the possibility of identifying other issues or finding different perspectives.

Both model approaches were easy to apply given the experience of analysts and later discussion with the SMEs indicated that the models were in accordance with their perceptions. As used here, these models derive from expert (analyst is an expert in the domain) rather than the facilitated (analyst supports the process) mode (Franco and Montibeller 2010). Nevertheless, there seems no reason why either of these methods cannot be used in a facilitated discussion as drivers to develop models. The particular difficultly present in this example (the need for a red team) may be less of a concern for the functional approach as this is more concerned with generalities rather than specific actions.

7.2 Validity

It is often noted that ‘all models are wrong, but some are nevertheless useful’ (Box 1979) and thus the models need to be considered for satisficing the notions of fitness for purpose rather than an absolute determination. Our feeling is that the models can be deemed valid if they represent the system under study well enough to be robust to the process of issue identification without recourse to extensive revision and reformat. Similarly, the models may be considered valid if they are accepted by the client group as representing a system with which they are familiar (see next heading).

We note that the sequential model method, while showing many features, becomes complicated by including all possible combinations of actions in an attempt to be as comprehensive as possible. The general nature of the functional model would remove the need for this much detail, as was shown by the number of issues that were found by the two methods. However, explicit identification in the sequential model at least verifies that specific issue has been found. A consequence of this is that a new way of doing things is likely to require an extra line in the sequential model, but may be accommodated in the functional model as a variant. The other concern is the granularity of the models. The experience was that the sequential model could go into considerable depth on specific topics whereas the functional model covered a broad area.

Both models will suffer from reproducibility problems. This is likely to be less severe for the sequential analysis route as the method leads to a systematic recording of events. Differences are likely to occur however in the level of detail and these will depend on the background of the participants and any overall environmental influences (such as testing a possible option). For functional analysis, it is entirely possible that different analysts (or even the same analysts at a later date) will produce seemingly completely different interpretations. The guidelines proposed here (e.g. identifying a higher level goal) may help to alleviate this. However, it is equally possible that the same issues will still be captured even though the model was different, though this will need to be tested.

7.3 Client perception

Experience has shown that some stakeholders are comfortable with an influence-like diagram, while others prefer the linear-layout style of a sequential analysis. For instance, in one example looking at immediate ‘solutions’ to an operational threat, it is straightforward to play through an event using a sequential model. In contrast, the use of influence in longer-term capability development (Bilusic et al. 2012) has been well received.

The models will have differing aspects of credibility. The sequential one has the advantage of depth to show rigour, whereas the functional model will show the overall picture on a single page. Whatever the form of the model it is likely that a procedure and some evaluation criteria need to be established to seek acceptance of specific substrate models. Importantly, the models lend themselves to a ready description of the issues and will be presented in a relevant form whether for improving current practice (sequential) or identifying other options (functional).

7.4 Fitness for purpose

From the above discussion, it is likely that a functional method would be more appropriate for the needs phase while the sequential method would be more applicable for identifying issues to be addressed in the requirements phase. Similarly, current operationally focussed issues are more likely to be found using the sequential method, whereas the functional model would be more suited to the operational (campaign) level. While the current emphasis on identifying materiel acquisitions probably places the focus on the needs and requirements phases, findings from such studies would also be useful for procedural or organisational insights for in-service usage.

The detailed nature of sequential analysis modelling leads to ready identification of specific deficiencies and to judgements on the degree and importance of such cases. In contrast, the functional analysis only has limited ability in this respect.

The number of occurrences of certain items may be a means of prioritising though it should be emphasised that both methods are subjective and results may be biased by differing level of inspection of certain areas over others. There may also be cases of double counting. In addition, an inherent bias occurs in the sequential approach through a tendency to emphasise those items that can be readily expressed in terms of a measureable effect. This may result in some areas being covered in great detail with others modelled at only a higher level. This bias can be moderated by applying a judicious weighting system that rewards distinct usages, such as the number of activities an issue appears in. There are also semi-quantitative means that can be applied to influence diagrams to identify issues of more importance than others. Studies based on either of these methods should thus be couched in terms of priority lists backed by a semi-quantitative audit trail and narrative comment.

The two methods thus have their strengths and weaknesses and, in classic OR tradition, selection of the appropriate approach will depend on the specific needs of the study. While one or the other may be the more appropriate within the context of mixed methods, we believe that both methods should be undertaken if time permits so as to provide a more robust study, particularly if insights need to be made about priorities. In addition, both methods can be used to structure the study and to conduct the detailed analysis. Both methods may be used to generate and highlight issues which can be grouped to identify common elements. This is explicitly laid out for the sequential methods, while some experience is needed to use the influence diagrams. Because influence diagrams typically take a higher level view of the system, identification of specific deficiencies is not as straightforward as for the sequential analysis approach.

8 Conclusions

The concept of a ‘substrate’ model was introduced as a mechanism to guide an analytic investigation so as to ameliorate the risk of introducing biases or focussing on less relevant aspects of a problem space. To illustrate the use of a substrate model for guiding a combinatorial approach a specific military example, operations in an IED environment, was studied with a particular activity, VCP, used as a case study. The clear exposition of an IED system allowed a well-structured and auditable process whereby potential issues of concern were identified. With judicious use of tagging of these issues, groupings of similar topics were possible allowing a structured approach to science and technology (or other) interventions. By having a documented audit trail, it is possible to provide semi-quantitative insights into prioritisation of options to redress problems associated with the issues of concern.

The material presented in this paper shows how a substrate model can be treated as an integral part of a ‘front-end’ study and provides a necessary component to the practice of problem structuring. In the paper, we propose a separation of the more objective aspects of a substrate model from the procedural aspects encompassed in the problems structuring process model.

The two types of substrate model provided different perspectives in deriving the issues of concern. While we have no evidence that the two methods will identify different issues, there is certainly an indication that there will be a difference in emphasis with the sequential model focussing on more concrete action items and the functional model favouring more general features such as support. Given this, we propose that the functional model is more appropriate to interventions at their earliest stage, while the sequential method is more suited to more developed management actions. We also stress that other forms of modelling may be used, but we are unable to comment if they will produce different insights. We do, however, contend that the style of model may well introduce bias in interpretation and that the analyst should be aware of this.

While this work has been directed towards a specific area of interest, namely a relatively new area of conflict where there is considerable uncertainty in both the blue and red actions, we feel that it has wider applications for studies where there are distinct processes and two or more discrete stakeholders. For instance, the health care/patient system could be examined as an information gathering exercise for longer-term strategies. This would contrast with optimisation studies where the issues are both well-known and quantifiable.