A method for extracting key performance indicators from maritime safety management norms

The maritime transportation industry currently employs several mandatory and non-mandatory norms of organizational safety management. These safety norms are commonly included in integrated maritime safety management systems, which aim at developing, monitoring, controlling and improving the safety of all related shipping operations. These systems are typically evaluated by following key performance indicators, which enable defined measures for various safety management components. However, the identification of indicators addressing safety management requirements constitutes a complex and generally unsystematic process for safety managers in the maritime industry. This article proposes a new method to assess the guidelines available in maritime safety management norms. The proposed method is applied to assess the content of two maritime safety management norms. The aim of this assessment is to identify a set of maritime safety management indicators that can systematically measure the most relevant components of maritime safety management. The application of this method resulted in the identification of 53 key performance indicators for monitoring and reviewing 23 identified safety management components that are commonly integrated into the functioning of maritime safety management systems. The method proposed provides guidance to accurately capture the actual aim and function of the key performance indicators. Furthermore, the indicators and safety components obtained with this method can be adopted as the basis for a safety management system and/or for the analysis of a safety management system already established in the industry. WMU J Marit Affairs (2016) 15:237–265 DOI 10.1007/s13437-015-0095-z * Osiris A. Valdez Banda osiris.valdez.banda@aalto.fi 1 Research Group on Maritime Risk and Safety, Department of Applied Mechanics, Aalto University, FI-00076 Aalto, Finland 2 Finnish Transport Safety Agency, Kumpulantie 9, 00520 Helsinki, Finland 3 Kotka Maritime Research Centre, Heikinkatu 7, FI-48100 Kotka, Finland


Introduction
The constant developments in maritime safety management originate from the need to reduce the risks associated with the development of maritime transport operations, the experience obtained from past accidents, the changing development of the global maritime traffic and the gradual adaptation of organizations to current maritime safety demands (Hänninen 2007;Jenisch 2004;Lappalainen et al. 2012;Reason 1997). Safety management aims at developing, planning, realizing and following organizational operations to prevent accidents and minimize risks related to the safety of people, property and the environment. However, due to the nature of maritime operations, establishing an efficient safety management system (SMS) and a desired safety culture represents a significant challenge in the industry (Lappalainen et al. 2011;Storgård 2012).
Achieving and maintaining a strong safety culture in the maritime industry is a complicated task which demands continuous improvement in the safety performance of all the stakeholders (Oltedal and Wadsworth 2010;Hetherington et al. 2006). The management of safety in maritime traffic operations is governed by the implementation of several norms and regulations. The International Maritime Organization (IMO) represents the main source of most existing regulations attempting to ensure and improve the safety performance of the global maritime industry (IMO 2015). In addition to the IMO, other organizations have also developed particular safety management norms applicable to certain types of maritime shipping sectors (see OCIMF 2008). This represents the existence of maritime shipping companies and complete shipping sectors with different levels of safety performance among the global shipping industry.
Commonly, the safety management of organizations is reviewed and evaluated by means of key performance indicators (KPIs) (Reiman and Pietikäinen 2012). In addition to measuring the current levels of operational and organizational safety, KPIs are also employed for tracking safety trends and developments (Øien 2001). The use of KPIs in collecting and analysing data and further in increasing the knowledge of the installed SMS is thus linked to the proactive approach to improving the technical and operational ship safety (Ek and Akselsson 2005).
Earlier studies (Øien et al. 2011a, b;Alwaer and Clements-Croome 2010;Hopkins 2009;Grabowski et al. 2007) have proposed several methodologies for identifying, assessing and selecting safety management KPIs. However, these methodologies are limited to measuring specific operations with no option to adopt and consider the actual safety management practices implemented in the organizations (Reiman and Pietikäinen 2012). Moreover, these methodologies have not explicitly included or analysed the existing requirements in safety management norms, which constitute the main guidelines for establishing, maintaining and improving any SMS. Thus, the lack of correspondence between measurements to analyse the system and the safety management norms implemented in the organization generates problems such as excessive bureaucracy, duplication of work, confusion between the requirements of the implemented norms and measuring organizational safety aspects without a clear understanding of the norm (Jørgensen et al. 2006). In the maritime sector, safety KPIs have also been poorly followed, necessitating more effective approaches to identifying, understanding and employing KPIs (Fälth and Ljungqvist 2013).
This study proposes a new method for performing a systematic evaluation of the requirements contained in maritime safety management norms, with the aim of providing appropriate guidance for employing the full potential of the norms and simultaneously combining the function of the norms more effectively. For this purpose, the realist evaluation methodology by Pawson and Tilley (1997) is applied to analyse the content of two maritime safety management norms. The aim is to construct a method for developing a proper interpretation, evaluation and measurement of the requirements contained in the norms to create a systematic adoption and measurement of these requirements within the actual functioning of the SMS. Thus, the obtained results enable the identification of specific KPIs for measuring the performance of any maritime SMS while complying with the requirements of the norms. Finally, the application of the method has also helped identify certain groups of components commonly interacting in the development of maritime safety management.
The rest of the article is organized as follows. Section 2 describes the theoretical foundation of the main concepts in this analysis. Section 3 presents the data and method of analysis implemented in this study. Section 4 presents the results to be obtained from the execution of the proposed method. Section 5 discusses the research findings, and Section 6 provides the final conclusions and recommendations for further research.
2 Theoretical foundation 2.1 Safety management perspective The management of safety is a shared responsibility among all the stakeholders in a company or a complete community. Initially, top-level management has the main responsibility for setting the basis of the safety culture, elaborating and establishing a safety policy, and determining the structure for controlling safety (Grote 2011;Leveson 2011). Subsequently, the safety responsibilities are shared and specifically delegated among middle management. Finally, the complete flow of the actual implementation of the safety practices is gradually reflected in the collective and individual performance of each person in the organization. Thus, from bottom up, every employee has to adjust their tasks to satisfy the safety regulation (Hollnagel 2014).
In line with this, the following safety management perspective, i.e. the systematic approach to understanding and describing safety management, is adopted: Safety management includes the arrangements made by the organisation to establish and promote a strong safety culture while achieving and controlling a determined safety performance. The detailed aim is to develop, plan, realize and follow operations to prevent accidents and minimize the risks related to the safety of people, property and the environment.

Safety management system
Safety in general is a system property, which has to be controlled at the system level (Leveson 2011). In this respect, an SMS, commonly established in each organization, plays a key role in developing and controlling the safety performance of the operations. Therefore, an SMS is established to support standardization of procedures, designs, trainings and the general responsibilities within the organization. Thus, the SMS has to be able to capture, understand and track norms and regulations applicable to the organization while supporting the monitoring compliance and the tracking of deviations with the aim of preventing accidents (Dekker 2014).
For the purposes of this study, an SMS is described as follows: The framework installed in an organisation that is composed of diverse and specialised components aiming to develop, plan, control, measure and analyse the safety management and safety performance of the organisation.

Safety key performance indicators
The literature divides KPIs into so-called leading and lagging indicators. Leading KPIs refer to measures for continuously monitoring identified inputs, which are needed to achieve a planned safety target and/or objective (Reiman and Pietikäinen 2012). Lagging KPIs are measurements that perform reactive monitoring to identify, e.g. when a planned objective or target has not been reached (Øien et al. 2011a). In this study, the KPIs measuring safety management are categorized into three groups based on Reiman and Pietikäinen (2012): Drive indicators: applied to change, maintain and reinforce different elements of the system. Their main function is to direct the socio-technical aspect of the system by motivating certain safety-related activities. Monitor indicators: implemented for monitoring the function of the system including but not limited to the efficacy of the control and development measures. These indicators measure the internal dynamics of the socio-technical system and provide information on system activities. Outcome indicators: these reflect a temporary end result of a process and/or an activity. These indicators focus on the result or consequence of the main tasks or processes in the organization.
3 Data and method of analysis 3.1 Safety management specifications

The ISM Code
In maritime safety management, the most important actor is the International Maritime Organization (IMO) with its International Safety Management Code (the ISM Code) introduced in 1994. The ISM Code aims at ensuring Bsafety at sea, preventing human injury or loss of life, and avoiding damage to the environment, in particular to the maritime environment and to property^ (IMO 1993). It includes 12 functional areas (listed in Table 1) for implementing the safe management and operations of ships and for preventing damage to the environment. These areas represent the main components for establishing an adequate SMS in the daily operations of any shipping company (Anderson 2003). Currently, the code is mandatory for all operative ships from 500 gross tonnages upwards.
The ISM Code has been written in a rather general way (Rodriguez and Hubbard 1998), and it allows much freedom in the implementation of the SMS in practice (Batalden and Sydnes 2014). The code is considered to consist of the essential and minimum requirements for establishing a maritime SMS. However, the advantages and complete potential of the available guidelines in the code have been found to be underestimated in the shipping industry (Schröder-Hinrichs 2010). This potential underestimation of the code's complete scope, together with the lack of a significant review and update of the code's content, have questioned the code's true type of approach (reactive or proactive) to safety management (Schröder-Hinrichs et al. 2013).

TMSA
The Tanker Management Self-Assessment (TMSA) by the Oil Companies International Maritime Forum (OCIMF) is a guide targeted to oil vessel operators for assessing, measuring and improving their SMS. The need for the TMSA rises from the lack of more detailed safety requirements for the vetting inspections performed by oil majors (Turker and Er 2008). The TMSA was originally implemented to ensure consistency between the international maritime conventions and the best practices within maritime oil transportation (OCIMF 2014). The TMSA encourages vessel operators to assess their SMS with a defined set of KPIs and provides best practice guidance for ensuring safety management (OCIMF 2008).
In this study, the TMSA is included due to its comprehensive safety management content and a list of general KPIs for measuring the safety management elements  Table 1) for the self-assessment of the SMS established by any oil tanker operators. These elements have been further extended into 22 sub-elements. Each sub-element has a set of KPIs integrated into four different stages, which describe the safety management level of the organization.
Stage 1 represents the basic requirements for the safety management established in the ISM Code. The aim of this stage is to ensure that the initial approach to seeking a constant improvement in safety management is represented by the strong initial link between the TMSA and the ISM Code. Moreover, this link demonstrates the commitment of an organization to effectively apply the code and therefore comply with the minimum demands for establishing an SMS.
Stages 2, 3 and 4 aim to continuously develop the safety management performance of oil vessel operators in order to demonstrate the companies' continuous improvement. The functional aim of these stages is to compare the actual safety management practice of an organization against the corresponding best practice guidance provided in the content of the TMSA.

The realist evaluation: aim and principles
Realist evaluation formulates a theory-driven evaluation of programmes. These programmes are associated with a structured hypothesis about social improvement (Pawson and Tilley 1997). Thus, programmes are basically assessed through the success or failure of the hypothesis. Therefore, programmes need to be reviewed among different stages in their applicability, from conceptualization and adaptation to application.
In the context of safety management, the hypothesis of social improvement is represented by the general aim of a safety management norm, which is to ensure and improve organizational safety. These programmes (norms) begin in the mind of policy architects and subsequently in the hands of the actual practitioners, thus inducing changes in conditions and behaviours within the socio-technical system where they are applied. Therefore, the applicability of these programmes can be ensured by the socio-technical systems and programme functionality being accompanied by several anticipated and unanticipated events that change over the duration of their application.
Considering the programme characteristics mentioned, the intention of the realist evaluation is to realistically and constantly assess how the programmes are supposed to function and how efficient their functioning is. For this purpose, realist evaluation implements the context-mechanism-outcome (CMO) analyses: & Context: assesses and describes the conditions in which the programmes are introduced and applied. & Mechanisms: corresponds to the resources and practical applications that make the programmes work. & Outcome: represents the analysis of both intended and unintended consequences derived from the programme implementation.
In essence, the realist evaluation seeks answers to BWhat works for whom in what circumstances in what respects and how?^ (Pawson and Tilley 1997). The evaluation can support the development of a structured and sustained method for the assessment of the hypothesis included in the programmes and the testing of the data derived from the assessment. Previously, realist evaluation has been adopted e.g. in social context analysis (Kazi 2003) and the assessment of health and medical services (Marchal et al. 2010). The realist evaluation is composed of four research stages: I. An establishment and formalization of the programme theories to be tested. In this study, this refers to the scope of the safety management norms. II. Collection of information to assess the hypothesis of the programmes. In this study, this is executed by collecting data from the analysis of the requirements included in the norms. For this purpose, expert consultations are performed (see Section 3.2.2). III. The execution of a systematic test based on the CMO process. In this study, the CMO is applied for the analysis of the safety management requirements contained in the two adopted norms. More specifically: & Context is linked to the way the requirements are subjected to the reasoning and environment of the affected organization. & Mechanisms are associated with the practical arrangements executed for developing all aspects planned in an SMS. This has to consider the way the organization uses the resources to make the system functional and supportive to obtain the planned objectives. & Outcome represents predefined estimations of all possible consequences arising from the application of these norms, and how the requirements of these norms need to be adapted to the plans, procedures and work processes within an SMS. Table 2 presents a set of general questions that can be addressed when applying the realist evaluation to assess the requirements of the maritime safety management norms. These questions anticipate the general and particular aspects of the system controlled and governed by the functional requirements of the norms.
IV. The assessment and interpretation of the CMO process. In this study, this stage aims to interpret the assessment of the norms and determine a set of KPIs, which represent a grounded measurement of the requirements in the norms and the complete function of the SMS. Figure 1 presents a general description of the four research stages of the realist evaluation as applied in this study.

The realist evaluation for identifying and selecting safety management KPIs
The implementation of the realist evaluation provides the foundation for identifying and selecting KPIs. The identification and selection of the KPI candidates has been performed by conducting a four-step process: Step one. Collection of maritime KPI candidates by consulting two information sources. First, the literature review of safety performance indicators for maritime safety management presented by Jalonen and Salmi (2009). This report reviews the ways safety management is monitored in different modes of maritime transportation and other relevant industries, e.g. the offshore industry. The second document is the shipping KPIs standard V2.0 (Shipping KPI 2013). The standard provides detailed measurement proposals for analysing several fields integrated into the shipping industry. In this step, one expert with over 10 years of experience in researching and practicing the management of safety in the Finnish maritime community has been consulted to review the relevancy of the selected KPI candidates for the analysis of the demanded safety management requirements.
Step two. KPIs for the requirements included in the content of the analysed norms are selected. The selection of the most relevant and final KPIs is conducted by applying the CMO queries presented in Table 1. These queries were assessed Table 2 The realist evaluation applied to the requirements of the safety management norms

Realist evaluation
Aspects to assess in the norm's requirement

Context
What is (are) the main organizational aspect(s) analysed by the requirement? Which is (are) the current task(s) developed in the organization linked to the requirement? What is the status of the organizational conditions influenced by the requirement? What and who are responsible for ensuring the requirement implementation and its maintenance? What is the current link of the requirement with similar requirement(s) in other implemented norms?

Mechanism
Which are the main organization's means for the implementation? How is the requirement currently communicated inside and outside of the organization? How is the organization able of ensuring the understanding of the importance of the requirement? How are the skills and capabilities of the responsible person(s) evaluated? How is the organization able of ensuring the link of the requirement with other implemented norms and regulations?

Outcome
What is the current level of fulfilment of the requirement? What are the expected results derived from the application of the requirement? What are the possible negative aspects (internal and external) that could affect before the implementation of the requirement? What kind of improvement can be obtained after implementing the requirement? & Two researchers with more than 5 years of experience from maritime traffic safety & Three researchers with more than 4 years of experience from port logistics & A researcher with more than 5 years of experience from marine environmental protection & A maritime safety consultant with over 8 years of experience on the field (the workshop moderator) & A maritime traffic safety researcher with 10 years of practical navigation experience & A researcher with more than 5 years of experience from maritime occupational safety and naval architecture Step three. A final review of the selected KPIs is performed in order to identify the links between the content of the utilized data sources and to avoid redundancy in the proposed measurements.
Step four. A classification of the selected KPIs by leading and lagging indicators, and in terms of drive, monitor and outcome indicators (Reiman and Pietikäinen 2012) introduced in Section 2.3. The aim is to clearly define the function of the different KPIs in the SMS.

Selecting the main components of maritime safety management
Once KPIs for all the requirements within the safety management norms are selected, the main maritime safety management components are defined, for which the KPIs provide information. The aim is to define the main components of maritime safety management based on the KPIs selected. For this, the characteristics of the KPIs are individually analysed by the experts who participated in the workshop presented in Step 2 of Section 3.2.2. In order to identify these components, the experts are provided with a simple questionnaire in which, after the description of each selected KPI, they are asked to define a specific safety management component that such a KPI directly measures. Thus, the instructions provided for performing this exercise included the following general requirement: Experts select a component by considering all aspects contained in the utilized specifications sources (Section 3.1). As the ISM Code is mandatory on board the majority of internationally trading ships, experts must specifically take eleven functional areas (2-12) of the ISM Code (see Table 1) into account. However, considering that the code represents the minimum demands for maritime safety, the experts are encouraged to provide extra components which are addressed in the documents but which are not explicitly regarded as functional areas in the code. To guide the experts in performing this task, a simple example is provided: Example. The KPI BPercentage of safety training received by masters, deck officers and seafarers^resulted from the analysis of the functional requirement number 6 (Resources and personnel) included in the ISM code. However, the main function of the KPI is linked to the relevancy of personnel's safety training. Thus, the impacted safety management component is training (for safety purposes).

Maritime safety management KPIs selected
Step one of the realist evaluation application resulted in 183 potential KPIs for the functional requirements of the ISM Code and 322 KPI candidates for the elements of the TMSA. Once each requirement included a set of KPI candidates, step two of the evaluation was executed. As it is not practically possible to describe how the realist evaluation was performed for all of these, the rest of the results section focuses on three examples (Tables 3, 4 and 5). These tables present the result from applying the evaluation methodology to assess the safety management requirements and select their corresponding KPIs.
In this first example, the adopted KPI is the number of company's safety and environmental policy reviews performed within a year. This KPI resulted after applying the general evaluation of the several factors addressed by the CMO questions. In Table 3, a justification of how the analysis of these questions supports the selection of KPI(s) is also described after analysing the questions and their respective answers.
In the second example, two KPIs are adopted to review and measure the analysed specification: first, the percentage of people trained with a training derived from accidents, incidents and near-misses reported, and second, the number of training sessions provided to ensure and improve safety on board ships.
In the final example, also two KPIs were adopted: first, the number of incidents reported within a period of time, and second, the number of systems controls detected as inefficient to prevent an occurred accident. Table 6 presents the complete list of safety management KPIs that, after the realist evaluation, are adopted for the analysed safety management requirements. Figure 2 presents the categorizations of the adopted KPIs based on the indicator types proposed in Reiman and Pietikäinen (2012). The figure presents a general classification between leading and lagging indicators, and the clustering of these indicators into drive, monitor and outcome KPIs.

Identified maritime safety management components
Extending the analysis of KPIs adopted after the application of the realist evaluation enabled identifying the main components of maritime safety management. Thus, this analysis identified 23 maritime safety management components, The organization's initial position for the management of safety in the organization.
This position is constantly evaluated in the periodical reviews of the established policy.
What are the tasks linked to the application of this requirement.
The complete SMS is influenced and ruled by the policy.
The goals reached and unreached in different areas of the SMS should influence the periodical updates of the policy.
What is the status of the main conditions in the organization for implementing the policy?
Organizational safety position, resources, politics and structure for policy implementation.
The safety management position is expressed in the policy and it is assessed against the general conditions of the organization. This is performed in the policy review.
What and who are responsible for the policy implementation and maintenance?
Commonly, the company and the safety designated person ashore (DPA).
This responsibility is tested in the company's safety and environmental policy review.
What is the current link of the requirement with other norms and regulations?
An integrated safety policy should cover all the demands specified in the norms applied in the organization.
This important link is tested in the review of the policy. It reveals the actual commitment to the organizational management of safety.
Mechanism Which are the main means for the implementation?
These means consist of time and personnel needed to establish and maintain the policy. Furthermore, monetary resources for diffusion.
The evaluation of efficiency, about the utilized means to implement the policy, has to be performed in the reviews of the safety policy.
How is the requirement currently communicated inside and outside of the organization?
Several means of communication (safety meetings, via web, emails and posters).
The methods for diffusion of the policy should be also evaluated in its periodical review.
How is the organization capable of ensuring the understanding of the importance of the requirement?
The organization is responsible for testing and ensuring that personnel understand and apply the policy in their daily tasks.
As part of the policy periodical reviews, a review of the understanding and application of the policy at all levels in the organization has to be performed.
How are the skills and capabilities of the responsible person(s), for the functioning of the requirement, evaluated?
Periodical tests, reviews and audits.
The aspect should be also included or tested in the policy review.
presented in Table 7. In this list, the safety management functional requirements and elements included in the ISM Code and TMSA can be identified. Additionally, the list includes other components that, according to the experts, are needed for implementing, developing and evaluating maritime safety management. The list includes specific components such as management commitment, planning, training, communication, personnel awareness and involvement, noblame culture, the safety management IT system and the follow-up of corrective and preventive actions. Some of these components are also identified in previous studies of safety management in different industries such as (Ostrom et al. 1993;Budworth 1997;Flin et al. 2000). Safety policy is a general requirement in almost all safety management norms. Thus, the policy review has to cover all the demanded requirements included in the integrated standards.
Outcome What is the current fulfilment level of the requirement?
An organization has to provide a representation of the current level of the functioning of the safety policy. This representation serves as index of the system's current output and also as outcome to be utilized in the assessment of the commitment to continuous improvement.
The conclusions from the performed policy reviews should provide a certain index level describing the efficiency of its implementation. This is represented by the measuring and reporting of a determined KPI.
What are the expected results derived from the implementation of the policy?
The results are expected to confirm whether the policy is actually efficient.
The policy is reviewed together with the organization's objectives.
What are the possible negative aspects that could affect the requirement implementation?
Lack of organizational commitment to safety, lack of resources (e.g. communication means) and/or the establishment of an inappropriate/ unsuitable policy.
The review of the policy has to identify possible aspects affecting the policy implementation (areas of improvement).
What kind of improvement can be obtained after policy's performance evaluation?
The outcome of this evaluation is expected to be aligned with the general improvement of the SMS.
The conclusions of the policy review reveal if the policy is efficient and the aspects to be corrected, improved and re-planned. Each company demonstrates its position to improve safety skills and emergency reaction by ensuring personnel safety skills and capabilities with the provision of training.
The percentage of people trained with training derived from accidents, incidents and near misses reported and the number of training provided to ensure and improve safety on-board ships state a reactive and also proactive position for ensuring personnel safety skills.
Which is (are) the current task(s) developed in the organization which is (are) connected with the training for emergency response?
All tasks performed in the organization have to be linked to the preparation for emergency situations.
For this reason, the percentage of people trained with the lessons learned from the analysis of accidents, incident and near misses and the number of people trained for improving safety onboard are two indicators which aim at covering the complete personnel in the organization.
What is the status of the main conditions in the organization for implementing the planned emergency response?
These conditions represent the amount and quality of the training provided for emergency preparedness.
The number of training provided to ensure and improve safety on-board ships generates a needed index to assess the current conditions and the implemented actions.
What and who are responsible for ensuring the provision of emergency training?
Organization top management and their designated responsible (e.g. captains, DPAs and officers).
The indexes obtained from the review of the total personnel with training for emergency response in the organization, are an indication of the commitment from the person(s) responsible.
What is the current link of the requirement with similar requirement(s) in other implemented norms?
The training provided for emergency response should cover all the demands specified in the norms applied in the organization.
The training provided to ensure and improve safety on-board ships has to demonstrate its connection with the demanded requirements of the applied norms and regulations.
Mechanism Which are the main organization's means for the implementation of emergency training?
These are specified by the invested money, time and personnel available for emergency preparedness.
The amount of training provided a reflection of the invested resources for emergency preparedness, How is the requirement currently communicated inside and outside of the organization?
The communication approach is built based on the organization's plan and strategy and the current safety level.
The amount of training provided is the result of a coordination between plans and available resources. The programmed training has to be always communicated inside and outside of the organization.
How is the organization capable of ensuring the understanding of the importance of the emergency training and preparedness?
Each training provided for the organization's personnel should clearly specify its relevancy for the organizational SMS.
The reported indexes of training provided have to be aligned to the needs and safety demands of the organization. Attempts to cover the majority of the personnel demonstrate the commitment to ensure safety in the organization.
How are the skills and capabilities of the responsible person(s), for the functioning of the requirement, evaluated?
Periodical tests, reviews and audits.
The training provided for the personnel in the organization is commonly reviewed and audited including several aspects such as strategy planning and efficiency of application.
How is the organization capable of ensuring the link of the requirement with other implemented norms and regulations?
The functioning of the SMS has to adopt all demands on the applied regulations. The organization top management and safety responsible have to ensure an appropriate link between the requirements.
The complete amount of provided training has to be justified by the needs of the integrated SMS which covers all the demands on the applied norms and regulations.

Outcome
What is the current level of fulfilment of the requirement?
An index of the current training level in the organization has to be periodically stated in order to detect new training needs.
The number of training provided to ensure and improve safety on-board ships and the one detected from accidents, incidents and near misses have to be planned based on the assessment of requirement fulfilment.
What are the expected results derived from the application of the requirement?
These expected results should be specified in the planning phase for The number of training provided to ensure and improve safety on-board ships represents an After the identification of these safety management components, explicit descriptions of the following 23 components have been created: 1. Safety and environmental policy: as the initial step of safety management, the policy provides a general picture of the values and the main aim of the SMS applied in the organization. 2. Management commitment: represents the commitment from top management to effectively install and develop an SMS. 3. Company responsibilities and authority: establish the compromise, level of authority, and scope of organizational safety management for several participants. 4. Master's responsibilities and authority: articulately defines the master's obligations and level of authority within the SMS. 5. Resources and personnel: specifies the resources and personnel available to perform the operations of the SMS. 6. Designated persons: represents the selection of adequate personnel to ensure the link between the safety management of the company and operative personnel on board the ships. 7. Planning: a phase which sets the different targets of the operations within the SMS, and the necessary processes and programmes to efficiently obtain these targets. 8. Maintenance of the ship and equipment: the provision of the resources and personnel to ensure an effective maintenance supply for supporting the development of ship operations. indicator which is linked to the plans for improving the response capability.
What are the possible negative aspects that could affect the requirement implementation?
One example could be the lack of reporting and analysing accidents, incidents and particularly near misses which may affect the efficient application of the requirements.
The number of training derived from accidents, incidents and near misses will provide information about the accident, incident and near miss reporting culture in the organization.
What kind of improvement can be obtained after implementing the requirement?
In general, a better performance of tasks and role in the organization and better functioning of the SMS.
The number of training provided has the aim of covering the detected areas of opportunity and to gradually improve the safety performance in the organization. Analysis performed for detecting safety control violations in the established SMS is an indicator which is able to identify the commitment of the organization to prevent accidents.
Which is (are) the current task(s) developed in the organization which is (are) connected with the requirement?
Incident and accident analysis provides information for all areas in the organization.
The number of incidents reported represents an informative index which provided information about certain needs in the organizational SMS.
What is the status of the main conditions in the organization for implementing accident and incident analysis?
The systems must have, by obligation, the implementation and follow-up of actions after an incident is reported.
The number of incidents reported represents an informative index which provided on details the type of incidents suffered in a certain period of time.
What and who are responsible for ensuring the requirement implementation and maintenance?
The designated safety personnel ashore and on-board have the responsibility of ensuring the functioning of the requirement.
Detecting the control violations of the system provided evidence of the efficiency in the performance of the responsible personnel.
What is the current link of the requirement with similar requirement(s) in other implemented norms?
This requirement basically feeds all other requirements applied within the organizational SMS.
Analysis performed for detecting safety control violations in the established SMS provided evidence of the performance of the SMS and the efforts to improve it.
Mechanism Which are the main organization's means for incident analysis?
These are represented by time, personnel and money designated for the task.
The combination of the two proposed indicators should demonstrate the efficiency level in the use of resources planned and invested in the task.
How is the requirement currently communicated inside and outside of the organization?
The importance of incident analysis is clearly expressed in the organization by using means such as safety meetings and safety bulletins.
The number of incidents reported has also to address their respective actions produced after the analysis, and these have to be informed at all levels in the organization. Each incident reported must be followed by a specific implemented action to correct and prevent a new incident. This action should be derived after the incident investigation and analysis.
How are the skills and capabilities of the responsible person(s), for the functioning of the requirement, evaluated?
Incident analysis requires appropriate training for its elaboration.
The number of incidents reported and its derived investigation evidence the skill levels of the personnel responsible for the analysis.
How is the organization capable of ensuring the link of the requirement with other implemented norms?
The elaboration of incident analysis has to be aligned to the demands specified in all the norms applied in the organization.
The two adopted indicators provided key information for all areas of the organizational SMS.

Outcome
What is the current level of fulfilment of the requirement?
This level is determined by analysing the current outcome obtaining from the incident analysis and its effect in the organizational SMS.
The number of incidents reported provided an index of the reporting culture in the organization.
What are the expected results derived from the analysis of nonconformities, incidents and accidents?
Planned objectives in this requirement can be aligned to, e.g. the expected number of incidents reported in a certain period of time and the real number of received.
The number of incidents reported provides a point of reference for elaborating this analysis.
What are the possible negative aspects that could affect the requirement implementation?
Incident analysis depends on the reporting culture of the organization. For example, a company may obtain a significant amount of reports but with a poor quality data, thus leading to a subsequently poor analysis.
The number of incidents reported has to be compared with the number of actions elaborated for those reported incidents and a subsequently measuring their efficiency.
What kind of improvement can be obtained after implementing the requirement?
Accident and incident analysis can avoid or prevent the occurrence of major accidents. This requirement has a key role within the SMS.
The number of incidents reported, their posterior analysis and implemented actions are key aspects influencing the organizational SMS in general.  9. Training: represents a standard tool, which ensures that personnel continuously receive the correct formation in order to ensure the effective performance of the SMS.  to ensure and promote organizational safety. 12. Personnel awareness and involvement: the contribution and participation of the personnel in organizational safety matters. 13. Emergency preparedness: the organizational level of preparation to respond to emergency situations. This includes plans and procedures to ensure the organizational preparedness in case of emergencies. 14. No-blame culture: aims to promote adequate and fair reporting and analysis of any event threatening organizational safety. 15. Management review: performs an analysis and evaluation of the SMS from the top management perspective. 16. Internal audit: performs a general self-assessment of the SMS. 17. External audit: a safety review performed by a third body, which has no direct stake in the functioning of the organization. A general analysis of the SMS from the perspective of an external entity. 18. Feedback: a common internal practice, which provides evidence of the follow-up to every aspect related to the SMS. This may include answers to accident and incident reports. 19. Safety management IT system: An electronic tool that provides all the necessary support to develop different activities within the SMS. 20. Report and analysis of non-conformities, incidents and accidents: provides reference to the safety reporting practices of the organization and the analysis of any type of safety deviations. 21. Follow-up of corrective actions: evaluates the status of the corrective actions that have already been instigated. 22. Follow-up of preventive actions: evaluates the status of the preventive actions that have already been instigated. The realist evaluation method offers several aspects that enable the execution of a complete and accurate review of the requirements and guidance contained in the safety management norms. The context-mechanism-outcome (CMO) process provides a more  accurate interpretation of the norms' requirements by incorporating an initial assessment of the common environment in which these are applied. Furthermore, this analysis is able to consider also the available resources and tools for applying the requirements and estimations of the possible effect on their application. For the purposes of this study, realist evaluation aims at performing relatively complex analyses, which include understanding how norms and their requirements actually work, exactly for whom the norms work, estimating the possible circumstances of the systems in which these requirements are applied, and anticipating the possible outcomes after the application of the requirements. The set of queries included in the CMO process (Table 2) establishes some key aspects that need to be assessed when interpreting and applying the requirements of safety norms. The implementation of these queries has focused on the evaluation of the requirements for identifying a measurement proposal capable of monitoring and reporting the functioning of the requirement.
The general aim of the realist evaluation has been incorporated to construct a complete method acquiring a systematic and insightful understanding of the guidance and requirement contained in safety management norms. The scope of application of this method can support the construction and installation of a new SMS and the evaluation of an established SMS.
The proposed method provides a systematic process for establishing actual integrated SMS, meaning that the method is able to incorporate different norms aimed at similar purposes. This is possible with thorough understanding of the purpose and function of the requirements and the conditions in which the requirement is applied. Safety management norms are produced by assembling the safety components defined, therefore the majority of the existing safety management norms present similar content. However, without a proper interpretation and understanding of the norm content and its connectivity with other implemented norms, the constructed integrated system can turn dysfunctional and create work overload or excessive bureaucracy.
The application of the proposed method hence offers several advantages. The main advantage is the availability of a tool that provides a more systematic process for understanding, adopting and reviewing safety management requirements applicable to shipping companies, maritime authorities and/or any international maritime organization. Another advantage is the opportunity to create a simpler and more reasoned SMS within the maritime domain. This creates an SMS with a more proactive approach to ensuring and improving the development of maritime operations.
The main limitation of the proposed realist evaluation approach for the analysis of safety management norms and/or a particular SMS concerns the resources (time and personnel) needed to perform the evaluation. However, the investment can subsequently improve the functioning of the safety management and the organization of the operations.

The KPIs of maritime safety management
The determination of KPIs based on requirements which are mandatory or widely adopted in the industry enable a more feasible monitoring, evaluating and directing of the implementation of these norms. Also, selecting KPIs after the application of the realist evaluation provides a more comprehensive set of safety performance measurements for an actual integrated system. Moreover, application of the realist evaluation facilitates comprehension of the complete scope covered and measured by the indicator.
The three examples presented in Tables 3, 4 and 5 represent the application of a method which is capable of defining KPIs. In Table 3, the KPI Bnumber of company's safety and environmental policy reviews performed within a year^evaluates the specific responsibilities and obligations linked to the implementation and maintenance of the policy. The application of the CMO process provides an explicit representation of the tasks included in the review of the safety policy. As a metric, the number of policy reviews performed per year represents the organization's particular consideration to keep the function of the policy updated.
In the example presented in Table 4, the KPI Bpercentage of people trained with a training derived from accidents, incidents and near-misses reported^produces a reference of the organizational efforts to provide training that considers the reported and identified deficiencies and areas of opportunity in the SMS. The KPI Bnumber of training sessions provided to ensure and improve safety on-board ships^monitors the current safety situation on board ships, and considers the available options inside and outside the organization to designate new training to support the safety processes and the efficient utilization of organization's resources for reaching the planned safety goals. This KPI can be applied from a proactive safety perspective, e.g. when influencing the organizational work processes for obtaining a determined safety level. Table 5 presents the analysis for selecting two KPIs. First, the KPI Bnumber of incidents reported in a period of time^represents a numerical index of the incidents reported and the incident occurrence frequency. This KPI also provides a reference of the safety aspects detected in the organization, which demand certain action to control and improve the safety performance of the organization. Second, the KPI Banalysis performed for detecting safety control violations in the established SMS^provides information about the system safety control areas, which demand immediate attention to maintain the proper functioning of the SMS.
In general, the assessment of any requirement with the proposed method enables the performance of a cyclic analysis, which supports the understanding and relevancy of the requirements in norms and regulations, and subsequently facilitates the adoption of KPIs for the requirement analysed. Thus, this cyclic process of analysis attempts to solve one of the most classic disadvantages of KPIs: the disability to accurately capture the quality of metrics and provoke an erroneous focus on numbers only (Fälth and Ljungqvist 2013).
Comparing the final 53 safety management KPIs adopted against the ones suggested in other studies such as (Øien et al. 2011a, b;Alwaer and Clements-Croome 2010;Hopkins 2009;Grabowski et al. 2007) shows that whereas some of the indicators are common, the number of KPIs presented in this study represent an extended and wellfounded offer of measurements which can also be differentiated by leading and lagging indicators and by drive, monitor and outcome indicators.
Thereby, the 53 KPIs selected contain drive indicators such as Bthe number of campaigns to promote a no-blame culture^(46). This is a leading indicator that evaluates and guides organizational safety management by promoting the importance of a no-blame culture in e.g. reporting incidents and accidents. The selected KPIs also include monitor indicators such as Bthe percentage of obtained improvement in external audit based on the previously planned^(42). This is also a leading indicator that can test the efficiency in the planning of safety based on the results of an external audit of the SMS. Finally, the 53 KPIs also include outcome indicators such as Bthe number of accidents reported per year^(29). This lagging indicator represents an unexpected and undesired consequence of the operations performed in the organization. In total, the application of the proposed method has identified a total of 22 drive indicators, 11 monitor indicators and 20 outcome indicators. Together these 53 indicators can monitor, report and assess any maritime SMS based on the norms assessed in this study.
Thus, the general application of the proposed method can be tailored by the particular needs of an organization. This can either mean seeking KPIs which provide the necessary input to guide the SMS and its components in the correct direction for reaching the planned safety objectives, or determining KPIs which monitor and report the performance of the SMS. Furthermore, the KPIs can be employed in groups for the analysis of specific safety components, in which an organization may be interested. Thus, an organization is free to decide on a categorisation or level of importance of each KPI. In this article, the application of the proposed method is illustrated with the execution of the complete CMO process and evaluation of KPIs by a group of experts in maritime safety.

The identified components of maritime safety management
Although the proposed set of components should not be seen as the only acceptable way of dividing maritime safety management into smaller sub-areas, it can be regarded as a first step towards an integrative and comprehensive, yet rather simple description of the different aspects demanded in the implementation of the maritime safety management. The identified components in this study have already been used as a starting point in examining the inner mechanics of maritime safety management systems for key relevant stakeholders in the Finnish maritime industry. For instance, these are integrated into the Bayesian network model of maritime safety management presented in Hänninen et al. (2014).
The identification of the main components of maritime safety management is highly relevant to the management of safety because it clearly portrays all safety sub-areas integrated into the practical functioning of any maritime SMS. Although these identified components are expressed within the content of the norms and they are also well known by maritime safety practitioners, the fact that these are not considered the main functional areas of maritime safety management can result in a situation where their real importance and relevancy becomes under-represented.
Moreover, if the standards and guidelines are updated in the light of better system safety understanding, the sets of KPIs and safety management components could also be updated with the proposed methodology.

Validation of the analysis
The validation of the results (KPIs and safety management components) is performed by utilizing the construct validity process proposed by Trochim and Donnelly (2008). This process is employed to assess the adequacy of the obtained results for accurately representing the main aspects of maritime safety management. The validation consists of two aspects: translation validity and criterion-related validity.
In this paper, validation refers to translation validity, which focuses on the adequacy of the operationalisation of the obtained results. Translation validity includes face validity and content validity.
Face validity examines operationalisation and evaluates if the obtained results represent a good translation of the construct. For performing this validation, the results are presented to two safety-designated persons ashore (DPA), who have more than 20 years of maritime safety experience, including operational and administrative experience. The two experts agreed that the safety management components identified are essential for implementing and developing maritime operations. Also, the experts identified most of the selected KPIs as relevant measurements of maritime safety management.
Content validity reviews operationalisation against the relevant content domain for the construct. Initially, this validation indicated that the proposed components of maritime safety management are included in a common maritime SMS (see Lappalainen et al. 2012). Also, the operationalisation of safety management is similar to the functioning of e.g. quality and environmental management. Therefore, some of these components can be found in the traditionally called quality, safety and environmental (QSE) integrated management system (Celik 2009;Jørgensen et al. 2006;Reiman and Rollenhagen 2014). Furthermore, the consulted DPAs have assessed the applicability of the selected KPIs by comparing them against the KPIs employed in the SMS of their companies. Thus, the two experts mentioned that about 60-70 % of the adopted indicators are similar to the ones in their SMS, 15-25 % of these indicators are identified as new and interesting proposals, and 5-15 % of these indicators are identified as irrelevant for the analysis of their installed systems.
Finally, the reliability of the method proposed in this study can be tested by executing the same method in different shipping companies, maritime organizations and authorities established in different maritime contexts. The results obtained by applying this method may vary depending on the approach to analyse the system safety in those organizations. However, there should not be radical differences because maritime safety in the international maritime community is managed and regulated according to similar safety management norms.

Conclusions
This study presents a systematic method for understanding, analysing and applying the requirements demanded in the content of existing maritime safety management norms. This method is implemented to identify key parameters that need to be assessed for applying and/or measuring the requirements of these norms. Simultaneously, it anticipates and analyses the context (working environment and established safety culture) in which these are applied, reviews the main means and available tools for applying these requirements, and assesses the possible organizational safety outcome.
The article describes the application of a method for analysing the contents of the ISM Code and TMSA. The purpose is to identify a set of KPIs for the main components of maritime safety management. This has resulted in 53 maritime safety management KPIs and 23 safety management components relevant for the maritime industry. It can be concluded that this method provides detailed guidance for selecting safety KPIs which can accurately capture the actual aim and function of the indicator. Thus, it creates a basis for defining effective ways of measuring the maritime safety management and therefore introduces the initial input to create a well-founded SMS for any organization established within the maritime community that complies with the relevant norms.
The proposed method and the main findings of this research can also be applied to assess how the guidelines and requirements of an applicable safety norm are adopted and measured within a functional SMS. Thus, identifying whether the mandatory and voluntary safety norms adopted are appropriately integrated into the organizational safety control structure.