1 Introduction

The constant developments in maritime safety management originate from the need to reduce the risks associated with the development of maritime transport operations, the experience obtained from past accidents, the changing development of the global maritime traffic and the gradual adaptation of organizations to current maritime safety demands (Hänninen 2007; Jenisch 2004; Lappalainen et al. 2012; Reason 1997). Safety management aims at developing, planning, realizing and following organizational operations to prevent accidents and minimize risks related to the safety of people, property and the environment. However, due to the nature of maritime operations, establishing an efficient safety management system (SMS) and a desired safety culture represents a significant challenge in the industry (Lappalainen et al. 2011; Storgård 2012).

Achieving and maintaining a strong safety culture in the maritime industry is a complicated task which demands continuous improvement in the safety performance of all the stakeholders (Oltedal and Wadsworth 2010; Hetherington et al. 2006). The management of safety in maritime traffic operations is governed by the implementation of several norms and regulations. The International Maritime Organization (IMO) represents the main source of most existing regulations attempting to ensure and improve the safety performance of the global maritime industry (IMO 2015). In addition to the IMO, other organizations have also developed particular safety management norms applicable to certain types of maritime shipping sectors (see OCIMF 2008). This represents the existence of maritime shipping companies and complete shipping sectors with different levels of safety performance among the global shipping industry.

Commonly, the safety management of organizations is reviewed and evaluated by means of key performance indicators (KPIs) (Reiman and Pietikäinen 2012). In addition to measuring the current levels of operational and organizational safety, KPIs are also employed for tracking safety trends and developments (Øien 2001). The use of KPIs in collecting and analysing data and further in increasing the knowledge of the installed SMS is thus linked to the proactive approach to improving the technical and operational ship safety (Ek and Akselsson 2005).

Earlier studies (Øien et al. 2011a, b; Alwaer and Clements-Croome 2010; Hopkins 2009; Grabowski et al. 2007) have proposed several methodologies for identifying, assessing and selecting safety management KPIs. However, these methodologies are limited to measuring specific operations with no option to adopt and consider the actual safety management practices implemented in the organizations (Reiman and Pietikäinen 2012). Moreover, these methodologies have not explicitly included or analysed the existing requirements in safety management norms, which constitute the main guidelines for establishing, maintaining and improving any SMS. Thus, the lack of correspondence between measurements to analyse the system and the safety management norms implemented in the organization generates problems such as excessive bureaucracy, duplication of work, confusion between the requirements of the implemented norms and measuring organizational safety aspects without a clear understanding of the norm (Jørgensen et al. 2006). In the maritime sector, safety KPIs have also been poorly followed, necessitating more effective approaches to identifying, understanding and employing KPIs (Fälth and Ljungqvist 2013).

This study proposes a new method for performing a systematic evaluation of the requirements contained in maritime safety management norms, with the aim of providing appropriate guidance for employing the full potential of the norms and simultaneously combining the function of the norms more effectively. For this purpose, the realist evaluation methodology by Pawson and Tilley (1997) is applied to analyse the content of two maritime safety management norms. The aim is to construct a method for developing a proper interpretation, evaluation and measurement of the requirements contained in the norms to create a systematic adoption and measurement of these requirements within the actual functioning of the SMS. Thus, the obtained results enable the identification of specific KPIs for measuring the performance of any maritime SMS while complying with the requirements of the norms. Finally, the application of the method has also helped identify certain groups of components commonly interacting in the development of maritime safety management.

The rest of the article is organized as follows. Section 2 describes the theoretical foundation of the main concepts in this analysis. Section 3 presents the data and method of analysis implemented in this study. Section 4 presents the results to be obtained from the execution of the proposed method. Section 5 discusses the research findings, and Section 6 provides the final conclusions and recommendations for further research.

2 Theoretical foundation

2.1 Safety management perspective

The management of safety is a shared responsibility among all the stakeholders in a company or a complete community. Initially, top-level management has the main responsibility for setting the basis of the safety culture, elaborating and establishing a safety policy, and determining the structure for controlling safety (Grote 2011; Leveson 2011). Subsequently, the safety responsibilities are shared and specifically delegated among middle management. Finally, the complete flow of the actual implementation of the safety practices is gradually reflected in the collective and individual performance of each person in the organization. Thus, from bottom up, every employee has to adjust their tasks to satisfy the safety regulation (Hollnagel 2014).

In line with this, the following safety management perspective, i.e. the systematic approach to understanding and describing safety management, is adopted:

Safety management includes the arrangements made by the organisation to establish and promote a strong safety culture while achieving and controlling a determined safety performance. The detailed aim is to develop, plan, realize and follow operations to prevent accidents and minimize the risks related to the safety of people, property and the environment.

2.2 Safety management system

Safety in general is a system property, which has to be controlled at the system level (Leveson 2011). In this respect, an SMS, commonly established in each organization, plays a key role in developing and controlling the safety performance of the operations. Therefore, an SMS is established to support standardization of procedures, designs, trainings and the general responsibilities within the organization. Thus, the SMS has to be able to capture, understand and track norms and regulations applicable to the organization while supporting the monitoring compliance and the tracking of deviations with the aim of preventing accidents (Dekker 2014).

For the purposes of this study, an SMS is described as follows:

The framework installed in an organisation that is composed of diverse and specialised components aiming to develop, plan, control, measure and analyse the safety management and safety performance of the organisation.

2.3 Safety key performance indicators

The literature divides KPIs into so-called leading and lagging indicators. Leading KPIs refer to measures for continuously monitoring identified inputs, which are needed to achieve a planned safety target and/or objective (Reiman and Pietikäinen 2012). Lagging KPIs are measurements that perform reactive monitoring to identify, e.g. when a planned objective or target has not been reached (Øien et al. 2011a).

In this study, the KPIs measuring safety management are categorized into three groups based on Reiman and Pietikäinen (2012):

  • Drive indicators: applied to change, maintain and reinforce different elements of the system. Their main function is to direct the socio-technical aspect of the system by motivating certain safety-related activities.

  • Monitor indicators: implemented for monitoring the function of the system including but not limited to the efficacy of the control and development measures. These indicators measure the internal dynamics of the socio-technical system and provide information on system activities.

  • Outcome indicators: these reflect a temporary end result of a process and/or an activity. These indicators focus on the result or consequence of the main tasks or processes in the organization.

3 Data and method of analysis

3.1 Safety management specifications

3.1.1 The ISM Code

In maritime safety management, the most important actor is the International Maritime Organization (IMO) with its International Safety Management Code (the ISM Code) introduced in 1994. The ISM Code aims at ensuring “safety at sea, preventing human injury or loss of life, and avoiding damage to the environment, in particular to the maritime environment and to property” (IMO 1993). It includes 12 functional areas (listed in Table 1) for implementing the safe management and operations of ships and for preventing damage to the environment. These areas represent the main components for establishing an adequate SMS in the daily operations of any shipping company (Anderson 2003). Currently, the code is mandatory for all operative ships from 500 gross tonnages upwards.

Table 1 The functional areas of the ISM Code and the TMSA elements
Full size table

The ISM Code has been written in a rather general way (Rodriguez and Hubbard 1998), and it allows much freedom in the implementation of the SMS in practice (Batalden and Sydnes 2014). The code is considered to consist of the essential and minimum requirements for establishing a maritime SMS. However, the advantages and complete potential of the available guidelines in the code have been found to be underestimated in the shipping industry (Schröder-Hinrichs 2010). This potential underestimation of the code’s complete scope, together with the lack of a significant review and update of the code’s content, have questioned the code’s true type of approach (reactive or proactive) to safety management (Schröder-Hinrichs et al. 2013).

3.1.2 TMSA

The Tanker Management Self-Assessment (TMSA) by the Oil Companies International Maritime Forum (OCIMF) is a guide targeted to oil vessel operators for assessing, measuring and improving their SMS. The need for the TMSA rises from the lack of more detailed safety requirements for the vetting inspections performed by oil majors (Turker and Er 2008). The TMSA was originally implemented to ensure consistency between the international maritime conventions and the best practices within maritime oil transportation (OCIMF 2014). The TMSA encourages vessel operators to assess their SMS with a defined set of KPIs and provides best practice guidance for ensuring safety management (OCIMF 2008).

In this study, the TMSA is included due to its comprehensive safety management content and a list of general KPIs for measuring the safety management elements included. The TMSA contains 12 elements (shown in Table 1) for the self-assessment of the SMS established by any oil tanker operators. These elements have been further extended into 22 sub-elements. Each sub-element has a set of KPIs integrated into four different stages, which describe the safety management level of the organization.

Stage 1 represents the basic requirements for the safety management established in the ISM Code. The aim of this stage is to ensure that the initial approach to seeking a constant improvement in safety management is represented by the strong initial link between the TMSA and the ISM Code. Moreover, this link demonstrates the commitment of an organization to effectively apply the code and therefore comply with the minimum demands for establishing an SMS.

Stages 2, 3 and 4 aim to continuously develop the safety management performance of oil vessel operators in order to demonstrate the companies’ continuous improvement. The functional aim of these stages is to compare the actual safety management practice of an organization against the corresponding best practice guidance provided in the content of the TMSA.

3.2 Method of analysing the specifications

3.2.1 The realist evaluation: aim and principles

Realist evaluation formulates a theory-driven evaluation of programmes. These programmes are associated with a structured hypothesis about social improvement (Pawson and Tilley 1997). Thus, programmes are basically assessed through the success or failure of the hypothesis. Therefore, programmes need to be reviewed among different stages in their applicability, from conceptualization and adaptation to application.

In the context of safety management, the hypothesis of social improvement is represented by the general aim of a safety management norm, which is to ensure and improve organizational safety. These programmes (norms) begin in the mind of policy architects and subsequently in the hands of the actual practitioners, thus inducing changes in conditions and behaviours within the socio-technical system where they are applied. Therefore, the applicability of these programmes can be ensured by the socio-technical systems and programme functionality being accompanied by several anticipated and unanticipated events that change over the duration of their application.

Considering the programme characteristics mentioned, the intention of the realist evaluation is to realistically and constantly assess how the programmes are supposed to function and how efficient their functioning is. For this purpose, realist evaluation implements the context-mechanism-outcome (CMO) analyses:

  • Context: assesses and describes the conditions in which the programmes are introduced and applied.

  • Mechanisms: corresponds to the resources and practical applications that make the programmes work.

  • Outcome: represents the analysis of both intended and unintended consequences derived from the programme implementation.

In essence, the realist evaluation seeks answers to “What works for whom in what circumstances in what respects and how?” (Pawson and Tilley 1997). The evaluation can support the development of a structured and sustained method for the assessment of the hypothesis included in the programmes and the testing of the data derived from the assessment. Previously, realist evaluation has been adopted e.g. in social context analysis (Kazi 2003) and the assessment of health and medical services (Marchal et al. 2010). The realist evaluation is composed of four research stages:

  1. I.

    An establishment and formalization of the programme theories to be tested. In this study, this refers to the scope of the safety management norms.

  2. II.

    Collection of information to assess the hypothesis of the programmes. In this study, this is executed by collecting data from the analysis of the requirements included in the norms. For this purpose, expert consultations are performed (see Section 3.2.2).

  3. III.

    The execution of a systematic test based on the CMO process. In this study, the CMO is applied for the analysis of the safety management requirements contained in the two adopted norms. More specifically:

    • Context is linked to the way the requirements are subjected to the reasoning and environment of the affected organization.

    • Mechanisms are associated with the practical arrangements executed for developing all aspects planned in an SMS. This has to consider the way the organization uses the resources to make the system functional and supportive to obtain the planned objectives.

    • Outcome represents predefined estimations of all possible consequences arising from the application of these norms, and how the requirements of these norms need to be adapted to the plans, procedures and work processes within an SMS.

      Table 2 presents a set of general questions that can be addressed when applying the realist evaluation to assess the requirements of the maritime safety management norms. These questions anticipate the general and particular aspects of the system controlled and governed by the functional requirements of the norms.

      Table 2 The realist evaluation applied to the requirements of the safety management norms
      Full size table
  4. IV.

    The assessment and interpretation of the CMO process. In this study, this stage aims to interpret the assessment of the norms and determine a set of KPIs, which represent a grounded measurement of the requirements in the norms and the complete function of the SMS. Figure 1 presents a general description of the four research stages of the realist evaluation as applied in this study.

    Fig. 1
    figure 1

    Application of the realist evaluation to the maritime safety management norms (I–IV represent the described research stages mentioned in Section 3.2.1)

    Full size image

3.2.2 The realist evaluation for identifying and selecting safety management KPIs

The implementation of the realist evaluation provides the foundation for identifying and selecting KPIs. The identification and selection of the KPI candidates has been performed by conducting a four-step process:

  • Step one. Collection of maritime KPI candidates by consulting two information sources. First, the literature review of safety performance indicators for maritime safety management presented by Jalonen and Salmi (2009). This report reviews the ways safety management is monitored in different modes of maritime transportation and other relevant industries, e.g. the offshore industry. The second document is the shipping KPIs standard V2.0 (Shipping KPI 2013). The standard provides detailed measurement proposals for analysing several fields integrated into the shipping industry. In this step, one expert with over 10 years of experience in researching and practicing the management of safety in the Finnish maritime community has been consulted to review the relevancy of the selected KPI candidates for the analysis of the demanded safety management requirements.

  • Step two. KPIs for the requirements included in the content of the analysed norms are selected. The selection of the most relevant and final KPIs is conducted by applying the CMO queries presented in Table 1. These queries were assessed during a workshop with nine experts of maritime safety management in Finland. This group of experts consists of:

    • Two researchers with more than 5 years of experience from maritime traffic safety

    • Three researchers with more than 4 years of experience from port logistics

    • A researcher with more than 5 years of experience from marine environmental protection

    • A maritime safety consultant with over 8 years of experience on the field (the workshop moderator)

    • A maritime traffic safety researcher with 10 years of practical navigation experience

    • A researcher with more than 5 years of experience from maritime occupational safety and naval architecture

  • Step three. A final review of the selected KPIs is performed in order to identify the links between the content of the utilized data sources and to avoid redundancy in the proposed measurements.

  • Step four. A classification of the selected KPIs by leading and lagging indicators, and in terms of drive, monitor and outcome indicators (Reiman and Pietikäinen 2012) introduced in Section 2.3. The aim is to clearly define the function of the different KPIs in the SMS.

3.2.3 Selecting the main components of maritime safety management

Once KPIs for all the requirements within the safety management norms are selected, the main maritime safety management components are defined, for which the KPIs provide information. The aim is to define the main components of maritime safety management based on the KPIs selected. For this, the characteristics of the KPIs are individually analysed by the experts who participated in the workshop presented in Step 2 of Section 3.2.2.

In order to identify these components, the experts are provided with a simple questionnaire in which, after the description of each selected KPI, they are asked to define a specific safety management component that such a KPI directly measures. Thus, the instructions provided for performing this exercise included the following general requirement:

Experts select a component by considering all aspects contained in the utilized specifications sources (Section 3.1). As the ISM Code is mandatory on board the majority of internationally trading ships, experts must specifically take eleven functional areas (2–12) of the ISM Code (see Table 1) into account. However, considering that the code represents the minimum demands for maritime safety, the experts are encouraged to provide extra components which are addressed in the documents but which are not explicitly regarded as functional areas in the code. To guide the experts in performing this task, a simple example is provided:

Example. The KPI “Percentage of safety training received by masters, deck officers and seafarers” resulted from the analysis of the functional requirement number 6 (Resources and personnel) included in the ISM code. However, the main function of the KPI is linked to the relevancy of personnel’s safety training. Thus, the impacted safety management component is training (for safety purposes).

4 Results

4.1 Maritime safety management KPIs selected

Step one of the realist evaluation application resulted in 183 potential KPIs for the functional requirements of the ISM Code and 322 KPI candidates for the elements of the TMSA. Once each requirement included a set of KPI candidates, step two of the evaluation was executed. As it is not practically possible to describe how the realist evaluation was performed for all of these, the rest of the results section focuses on three examples (Tables 3, 4 and 5). These tables present the result from applying the evaluation methodology to assess the safety management requirements and select their corresponding KPIs.

Table 3 Application of the realist evaluation methodology to requirement 2.2 of the ISM Code: the company should ensure implementation and maintenance of the safety and environmental policy
Full size table
Table 4 Application of the realist evaluation methodology to aim 11 stage 2 of the TMSA: individuals are trained in their designated emergency response roles
Full size table
Table 5 Application of the realist evaluation methodology to requirement 9.1 of the ISM Code: the SMS should include procedures ensuring that non-conformities, accidents and hazardous situations are reported, investigated and analysed with the objective of improving safety
Full size table

In this first example, the adopted KPI is the number of company’s safety and environmental policy reviews performed within a year. This KPI resulted after applying the general evaluation of the several factors addressed by the CMO questions. In Table 3, a justification of how the analysis of these questions supports the selection of KPI(s) is also described after analysing the questions and their respective answers.

In the second example, two KPIs are adopted to review and measure the analysed specification: first, the percentage of people trained with a training derived from accidents, incidents and near-misses reported, and second, the number of training sessions provided to ensure and improve safety on board ships.

In the final example, also two KPIs were adopted: first, the number of incidents reported within a period of time, and second, the number of systems controls detected as inefficient to prevent an occurred accident. Table 6 presents the complete list of safety management KPIs that, after the realist evaluation, are adopted for the analysed safety management requirements.

Table 6 The KPIs selected to evaluate the norm requirements
Full size table

4.2 Categorisation of the maritime safety management KPIs

Figure 2 presents the categorizations of the adopted KPIs based on the indicator types proposed in Reiman and Pietikäinen (2012). The figure presents a general classification between leading and lagging indicators, and the clustering of these indicators into drive, monitor and outcome KPIs.

Fig. 2
figure 2

Classification of the 53 selected KPIs (adapted to the extended system model presented in Reiman and Pietikäinen (2012))

Full size image

4.3 Identified maritime safety management components

Extending the analysis of KPIs adopted after the application of the realist evaluation enabled identifying the main components of maritime safety management. Thus, this analysis identified 23 maritime safety management components, presented in Table 7. In this list, the safety management functional requirements and elements included in the ISM Code and TMSA can be identified. Additionally, the list includes other components that, according to the experts, are needed for implementing, developing and evaluating maritime safety management. The list includes specific components such as management commitment, planning, training, communication, personnel awareness and involvement, no-blame culture, the safety management IT system and the follow-up of corrective and preventive actions. Some of these components are also identified in previous studies of safety management in different industries such as (Ostrom et al. 1993; Budworth 1997; Flin et al. 2000).

Table 7 The safety management components identified and measured with the KPIs
Full size table

After the identification of these safety management components, explicit descriptions of the following 23 components have been created:

  1. 1.

    Safety and environmental policy: as the initial step of safety management, the policy provides a general picture of the values and the main aim of the SMS applied in the organization.

  2. 2.

    Management commitment: represents the commitment from top management to effectively install and develop an SMS.

  3. 3.

    Company responsibilities and authority: establish the compromise, level of authority, and scope of organizational safety management for several participants.

  4. 4.

    Master’s responsibilities and authority: articulately defines the master’s obligations and level of authority within the SMS.

  5. 5.

    Resources and personnel: specifies the resources and personnel available to perform the operations of the SMS.

  6. 6.

    Designated persons: represents the selection of adequate personnel to ensure the link between the safety management of the company and operative personnel on board the ships.

  7. 7.

    Planning: a phase which sets the different targets of the operations within the SMS, and the necessary processes and programmes to efficiently obtain these targets.

  8. 8.

    Maintenance of the ship and equipment: the provision of the resources and personnel to ensure an effective maintenance supply for supporting the development of ship operations.

  9. 9.

    Training: represents a standard tool, which ensures that personnel continuously receive the correct formation in order to ensure the effective performance of the SMS.

  10. 10.

    Ship operations: an important end point which reflects the results of the application of all resources and efforts invested in maritime safety management.

  11. 11.

    Communication: it represents the means and forms employed by the organization to ensure and promote organizational safety.

  12. 12.

    Personnel awareness and involvement: the contribution and participation of the personnel in organizational safety matters.

  13. 13.

    Emergency preparedness: the organizational level of preparation to respond to emergency situations. This includes plans and procedures to ensure the organizational preparedness in case of emergencies.

  14. 14.

    No-blame culture: aims to promote adequate and fair reporting and analysis of any event threatening organizational safety.

  15. 15.

    Management review: performs an analysis and evaluation of the SMS from the top management perspective.

  16. 16.

    Internal audit: performs a general self-assessment of the SMS.

  17. 17.

    External audit: a safety review performed by a third body, which has no direct stake in the functioning of the organization. A general analysis of the SMS from the perspective of an external entity.

  18. 18.

    Feedback: a common internal practice, which provides evidence of the follow-up to every aspect related to the SMS. This may include answers to accident and incident reports.

  19. 19.

    Safety management IT system: An electronic tool that provides all the necessary support to develop different activities within the SMS.

  20. 20.

    Report and analysis of non-conformities, incidents and accidents: provides reference to the safety reporting practices of the organization and the analysis of any type of safety deviations.

  21. 21.

    Follow-up of corrective actions: evaluates the status of the corrective actions that have already been instigated.

  22. 22.

    Follow-up of preventive actions: evaluates the status of the preventive actions that have already been instigated.

  23. 23.

    Documentation: represents the common practice of documenting all the generated information within the SMS.

5 Discussion

5.1 The application of the realist evaluation for assessing the requirements of maritime safety management norms

The realist evaluation method offers several aspects that enable the execution of a complete and accurate review of the requirements and guidance contained in the safety management norms. The context-mechanism-outcome (CMO) process provides a more accurate interpretation of the norms’ requirements by incorporating an initial assessment of the common environment in which these are applied. Furthermore, this analysis is able to consider also the available resources and tools for applying the requirements and estimations of the possible effect on their application. For the purposes of this study, realist evaluation aims at performing relatively complex analyses, which include understanding how norms and their requirements actually work, exactly for whom the norms work, estimating the possible circumstances of the systems in which these requirements are applied, and anticipating the possible outcomes after the application of the requirements.

The set of queries included in the CMO process (Table 2) establishes some key aspects that need to be assessed when interpreting and applying the requirements of safety norms. The implementation of these queries has focused on the evaluation of the requirements for identifying a measurement proposal capable of monitoring and reporting the functioning of the requirement.

The general aim of the realist evaluation has been incorporated to construct a complete method acquiring a systematic and insightful understanding of the guidance and requirement contained in safety management norms. The scope of application of this method can support the construction and installation of a new SMS and the evaluation of an established SMS.

The proposed method provides a systematic process for establishing actual integrated SMS, meaning that the method is able to incorporate different norms aimed at similar purposes. This is possible with thorough understanding of the purpose and function of the requirements and the conditions in which the requirement is applied. Safety management norms are produced by assembling the safety components defined, therefore the majority of the existing safety management norms present similar content. However, without a proper interpretation and understanding of the norm content and its connectivity with other implemented norms, the constructed integrated system can turn dysfunctional and create work overload or excessive bureaucracy.

The application of the proposed method hence offers several advantages. The main advantage is the availability of a tool that provides a more systematic process for understanding, adopting and reviewing safety management requirements applicable to shipping companies, maritime authorities and/or any international maritime organization. Another advantage is the opportunity to create a simpler and more reasoned SMS within the maritime domain. This creates an SMS with a more proactive approach to ensuring and improving the development of maritime operations.

The main limitation of the proposed realist evaluation approach for the analysis of safety management norms and/or a particular SMS concerns the resources (time and personnel) needed to perform the evaluation. However, the investment can subsequently improve the functioning of the safety management and the organization of the operations.

5.2 The KPIs of maritime safety management

The determination of KPIs based on requirements which are mandatory or widely adopted in the industry enable a more feasible monitoring, evaluating and directing of the implementation of these norms. Also, selecting KPIs after the application of the realist evaluation provides a more comprehensive set of safety performance measurements for an actual integrated system. Moreover, application of the realist evaluation facilitates comprehension of the complete scope covered and measured by the indicator.

The three examples presented in Tables 3, 4 and 5 represent the application of a method which is capable of defining KPIs. In Table 3, the KPI “number of company’s safety and environmental policy reviews performed within a year” evaluates the specific responsibilities and obligations linked to the implementation and maintenance of the policy. The application of the CMO process provides an explicit representation of the tasks included in the review of the safety policy. As a metric, the number of policy reviews performed per year represents the organization’s particular consideration to keep the function of the policy updated.

In the example presented in Table 4, the KPI “percentage of people trained with a training derived from accidents, incidents and near-misses reported” produces a reference of the organizational efforts to provide training that considers the reported and identified deficiencies and areas of opportunity in the SMS. The KPI “number of training sessions provided to ensure and improve safety on-board ships” monitors the current safety situation on board ships, and considers the available options inside and outside the organization to designate new training to support the safety processes and the efficient utilization of organization’s resources for reaching the planned safety goals. This KPI can be applied from a proactive safety perspective, e.g. when influencing the organizational work processes for obtaining a determined safety level.

Table 5 presents the analysis for selecting two KPIs. First, the KPI “number of incidents reported in a period of time” represents a numerical index of the incidents reported and the incident occurrence frequency. This KPI also provides a reference of the safety aspects detected in the organization, which demand certain action to control and improve the safety performance of the organization. Second, the KPI “analysis performed for detecting safety control violations in the established SMS” provides information about the system safety control areas, which demand immediate attention to maintain the proper functioning of the SMS.

In general, the assessment of any requirement with the proposed method enables the performance of a cyclic analysis, which supports the understanding and relevancy of the requirements in norms and regulations, and subsequently facilitates the adoption of KPIs for the requirement analysed. Thus, this cyclic process of analysis attempts to solve one of the most classic disadvantages of KPIs: the disability to accurately capture the quality of metrics and provoke an erroneous focus on numbers only (Fälth and Ljungqvist 2013).

Comparing the final 53 safety management KPIs adopted against the ones suggested in other studies such as (Øien et al. 2011a, b; Alwaer and Clements-Croome 2010; Hopkins 2009; Grabowski et al. 2007) shows that whereas some of the indicators are common, the number of KPIs presented in this study represent an extended and well-founded offer of measurements which can also be differentiated by leading and lagging indicators and by drive, monitor and outcome indicators.

Thereby, the 53 KPIs selected contain drive indicators such as “the number of campaigns to promote a no-blame culture” (46). This is a leading indicator that evaluates and guides organizational safety management by promoting the importance of a no-blame culture in e.g. reporting incidents and accidents. The selected KPIs also include monitor indicators such as “the percentage of obtained improvement in external audit based on the previously planned” (42). This is also a leading indicator that can test the efficiency in the planning of safety based on the results of an external audit of the SMS. Finally, the 53 KPIs also include outcome indicators such as “the number of accidents reported per year” (29). This lagging indicator represents an unexpected and undesired consequence of the operations performed in the organization. In total, the application of the proposed method has identified a total of 22 drive indicators, 11 monitor indicators and 20 outcome indicators. Together these 53 indicators can monitor, report and assess any maritime SMS based on the norms assessed in this study.

Thus, the general application of the proposed method can be tailored by the particular needs of an organization. This can either mean seeking KPIs which provide the necessary input to guide the SMS and its components in the correct direction for reaching the planned safety objectives, or determining KPIs which monitor and report the performance of the SMS. Furthermore, the KPIs can be employed in groups for the analysis of specific safety components, in which an organization may be interested. Thus, an organization is free to decide on a categorisation or level of importance of each KPI. In this article, the application of the proposed method is illustrated with the execution of the complete CMO process and evaluation of KPIs by a group of experts in maritime safety.

5.3 The identified components of maritime safety management

Although the proposed set of components should not be seen as the only acceptable way of dividing maritime safety management into smaller sub-areas, it can be regarded as a first step towards an integrative and comprehensive, yet rather simple description of the different aspects demanded in the implementation of the maritime safety management. The identified components in this study have already been used as a starting point in examining the inner mechanics of maritime safety management systems for key relevant stakeholders in the Finnish maritime industry. For instance, these are integrated into the Bayesian network model of maritime safety management presented in Hänninen et al. (2014).

The identification of the main components of maritime safety management is highly relevant to the management of safety because it clearly portrays all safety sub-areas integrated into the practical functioning of any maritime SMS. Although these identified components are expressed within the content of the norms and they are also well known by maritime safety practitioners, the fact that these are not considered the main functional areas of maritime safety management can result in a situation where their real importance and relevancy becomes under-represented.

Moreover, if the standards and guidelines are updated in the light of better system safety understanding, the sets of KPIs and safety management components could also be updated with the proposed methodology.

5.4 Validation of the analysis

The validation of the results (KPIs and safety management components) is performed by utilizing the construct validity process proposed by Trochim and Donnelly (2008). This process is employed to assess the adequacy of the obtained results for accurately representing the main aspects of maritime safety management. The validation consists of two aspects: translation validity and criterion-related validity.

In this paper, validation refers to translation validity, which focuses on the adequacy of the operationalisation of the obtained results. Translation validity includes face validity and content validity.

Face validity examines operationalisation and evaluates if the obtained results represent a good translation of the construct. For performing this validation, the results are presented to two safety-designated persons ashore (DPA), who have more than 20 years of maritime safety experience, including operational and administrative experience. The two experts agreed that the safety management components identified are essential for implementing and developing maritime operations. Also, the experts identified most of the selected KPIs as relevant measurements of maritime safety management.

Content validity reviews operationalisation against the relevant content domain for the construct. Initially, this validation indicated that the proposed components of maritime safety management are included in a common maritime SMS (see Lappalainen et al. 2012). Also, the operationalisation of safety management is similar to the functioning of e.g. quality and environmental management. Therefore, some of these components can be found in the traditionally called quality, safety and environmental (QSE) integrated management system (Celik 2009; Jørgensen et al. 2006; Reiman and Rollenhagen 2014). Furthermore, the consulted DPAs have assessed the applicability of the selected KPIs by comparing them against the KPIs employed in the SMS of their companies. Thus, the two experts mentioned that about 60–70 % of the adopted indicators are similar to the ones in their SMS, 15–25 % of these indicators are identified as new and interesting proposals, and 5–15 % of these indicators are identified as irrelevant for the analysis of their installed systems.

Finally, the reliability of the method proposed in this study can be tested by executing the same method in different shipping companies, maritime organizations and authorities established in different maritime contexts. The results obtained by applying this method may vary depending on the approach to analyse the system safety in those organizations. However, there should not be radical differences because maritime safety in the international maritime community is managed and regulated according to similar safety management norms.

6 Conclusions

This study presents a systematic method for understanding, analysing and applying the requirements demanded in the content of existing maritime safety management norms. This method is implemented to identify key parameters that need to be assessed for applying and/or measuring the requirements of these norms. Simultaneously, it anticipates and analyses the context (working environment and established safety culture) in which these are applied, reviews the main means and available tools for applying these requirements, and assesses the possible organizational safety outcome.

The article describes the application of a method for analysing the contents of the ISM Code and TMSA. The purpose is to identify a set of KPIs for the main components of maritime safety management. This has resulted in 53 maritime safety management KPIs and 23 safety management components relevant for the maritime industry. It can be concluded that this method provides detailed guidance for selecting safety KPIs which can accurately capture the actual aim and function of the indicator. Thus, it creates a basis for defining effective ways of measuring the maritime safety management and therefore introduces the initial input to create a well-founded SMS for any organization established within the maritime community that complies with the relevant norms.

The proposed method and the main findings of this research can also be applied to assess how the guidelines and requirements of an applicable safety norm are adopted and measured within a functional SMS. Thus, identifying whether the mandatory and voluntary safety norms adopted are appropriately integrated into the organizational safety control structure.

CMO:

Context Mechanism Outcome

DPA:

Designated Person Ashore

IMO:

International Maritime Organization

ISM Code:

International Safety Management Code

IT:

Information Technology

KPI:

Key Performance Indicator

OCIMF:

Oil Companies International Maritime Forum

QSE:

Quality Safety and Environmental

SMS:

Safety Management System

TMSA:

Tanker Management Self-Assessment