1 Introduction

The National Institute of Standards and Technology (NIST) Cyber Security Framework (CSF) [28] is a risk-based approach to manage risks organisations face from a cyber security perspective. Similarly, several frameworks such as NIST SP 800-53 [27], COBIT5 [17], ISO/IEC 27001:2013 [23], ISA 62443-2-1:2009 [21], and ISA 62443-3-3:2013 [22] are being used to assess cyber security risk from different perspectives and outcomes are measured using different yardsticks. Often, navigating the various frameworks can be challenging for organisations, especially if such expertise are not present internally. Given the rapidly changing technology and threat landscape, assessing the cyber security posture of an organisation, regardless of their business or size, is paramount.

Our focus of this paper is to demonstrate the application (Sect. 3) of NIST CSF in a local government organisation and provide recommendations (Sect. 5) based on our findings (Sect. 4).

The main contributions of this paper are:

  • The adoption of the NIST CSF as an Assessment Tool and targeting different levels of the organisation, depending on their level of expertise and job function to obtain responses to facilitate assessment.

  • Quantification of the assessment to reflect severity of actual risk, which in turn enabled the organisation to effectively address the issues to attain desired level of compliance.

  • A detailed review of similar frameworks used in the industry and relevant case studies (Sect. 6).

The next section provides a background of the NIST CSF and its components. We recommend the reader to refer to NIST [28] for additional details and strategies for suitable approaches to implement, which would vary from organisation to organisation.

2 The NIST CSF

The NIST CSF [28] consists of the Framework Core, the Framework Implementation Tiers, and the Framework Profiles. The Framework Core consists of five concurrent and continuous functions; Identify, Protect, Detect, Respond, and Recover. We designed an Assessment Tool for our investigation based on these functions, which provided a systematic approach to ascertain the organisations cyber security risk management practices and processes.

The Framework Implementation Tiers describe the level an organisations cyber security risk management practices that comply with the framework. Tiers provide context and degree to which cyber security risks are managed and extent to which business needs are considered in cyber security risk management. The Assessment Tool enabled the determination of the organisations Current Tier based on various internal and external factors such as their risk management practices, threat environment, legal and regulatory requirements, business/mission objectives, and organisational constraints. Organisations should also determine the Desired Tier, provided it is feasible to implement, reduces cyber security risks, and meets the organisational goals. The following are descriptions of the tier levels [28]:

  • Tier-1 (Partial): risk management practices are not formalised and managed in an ad hoc manner, lack awareness of cyber security risks organisation wide, and do not have processes in place to collaborate with external entities.

  • Tier-2 (Risk Informed): risk management practices are formalised but not integrated organisation wide, but cyber security activities are prioritised based on risks with adequate means to perform related duties, with informal means to communicate cyber security information internally and externally.

  • Tier-3 (Repeatable): risk management practices are formalised and policies are in place and are adaptable to cyber threats. Organisation-wide approach is required to manage cyber security with skilled and knowledgeable personnel to respond and understand dependencies and role of external partners.

  • Tier-4 (Adaptive): cyber security practices are based on lessons learnt and predictive indicators, with continuous improvement, adaptability, and timely response. Organisation-wide approach to manage cyber security risks is part of the organisational culture and actively shares with external partners.

The Framework Profile represents the outcomes based on the business needs the organisation characterised from the Framework Core and determined using the Assessment Tool. Consequently, a Current Profile (the “as is” state) and a Target Profile (the “to be” state) can be used to identify opportunities for improving the cyber security of the organisation [28]. Framework profiles can be determined based on particular implementation scenarios, and therefore, the gap between Current Profile and Target Profile would vary as per scenario. In this paper, a local government-specific approach to CSF was adapted. However, industry-specific tailoring may be performed for the CSF.

3 Methodology

The NIST CSF allowed us to design an Assessment Tool targeted at three levels of participants within the organisation, i.e. executive, management and technical. The rationale was to ascertain organisation-wide understanding of cyber security risks. Hence, the Assessment Tool comprised of questions addressing the requirements outlined as per the NIST CSF.

The questions were selected based on the nature and relevance to the level of participant. This is because the NIST CSF comprised of questions that were both technical and non-technical. Therefore, it would have been unrealistic to expect deep knowledge of technical operations or implementation level details from a policy level executive.

In order to assist us determine a baseline (i.e. the Desired Tier), additional questions were included in the Assessment Tool to determine the nature of the organisation and its business. This was then followed by the remaining requirements comprised in the NIST CSF.

3.1 Determining compliance

The compliance for each measure was based on the responses provided by the participants. They were graded as either, Complaint, Partially Compliant, or Non-Compliant; and each was assigned scores of either 10, 5, or 0, respectively, for each core function’s subcategory. Any subcategory that was not applicable depending on the Desired Tier level was excluded from the compliance score calculation.

Given the number of security requirements for each Core Function’s subcategory is N, then the number of applicable requirements in each subcategory given the Desired Tier level is \(N^{\prime }\). Therefore, the total compliance score C for each core function’s category can be defined as:

$$\begin{aligned} C=\frac{\varSigma {R}}{\varSigma {N^{\prime }\times 10}} \end{aligned}$$
(1)

where R is the compliance score for each category of the respective Core Function.

Additionally, a detailed document audit was conducted on existing policies and procedures. The Information Technology (IT) infrastructure (internal, remote locations, and cloud) were reviewed, and a detailed internal vulnerability assessment was also conducted during our investigation.

4 Findings

The responses provided by the Executive, Management, and Technical participants gave insight into the organisation’s cyber security posture. Table 1 shows the summary of the compliance of NIST CSF assessment. The compliance scores were determined based on Eq. 1 presented previously.

Table 1 NIST CSF compliance matrix
Full size table

For Identify core function, the organisation scored 36%. Their ability to track assets centrally, keep management informed, and understand operational risks from a cyber security perspective was limited, while a strategy to manage such risks did not exist. However, the organisation understood its business well and were able set priorities to support risk management decisions.

Access to physical/virtual assets were through authorisation and well-defined processes. The staff were trained and informed adequately of information security related duties and responsibilities. Certain aspects of data security related to confidentiality and availability were done reasonably well, however, assuring integrity of data needed improvement. Similarly, local maintenance and remote maintenance of IT infrastructure were carried out in a manner consistent to policies and procedures. However, relevant policies, processes, and procedures, as well as technology to assist the protection of information systems and relevant assets, were lacking. Therefore, in aggregate, the organisation scored 45% compliance for Protect core function.

The organisation scored weakest in the detection of cyber security incidents with a score of 25%. Although certain monitoring activities were in place to track physical security and malicious code, timely detection of anomalous activities and detection processes were lacking or non-existent.

Despite the lack of a specific response plan to respond to a cyber security events, the organisation had measures in place to report incidents and coordinate activities to respond adequately, which resulted in a 38% compliance score for Respond core function. These practices are updated from time to time; however, mechanism to perform post-incident analysis or to mitigate future cyber security events has not been implemented presently.

Interestingly, the organisation was well prepared to deal with recovery and resumption of core services after a cyber security event. The recovery plans in place are tested, updated, and improved periodically, thus receiving full compliance for Recover core functionality of the framework.

5 Recommendations

Based on the findings, the following recommendations were made with respect to each core function of the NIST CSF.

5.1 Identify

  1. (a)

    Establish a central inventory of assets, including physical devices and systems, software, and external systems with all required information and prioritise based on classification, criticality, and business value.

  2. (b)

    Identify the organisations role in the supply chain (i.e. producer-consumer model) as it captures and retains public data, collects revenue, and provides services to its stakeholders.

  3. (c)

    Establish an Information Security policy and reference relevant federal and state policies regarding cyber security to ensure legal and regulatory requirements are understood and managed.

  4. (d)

    Identify and prioritise threats and vulnerabilities, both internal and external, to determine cyber security risks to the organisations operations, assets, and individuals.

  5. (e)

    Establish risk management processes that are managed and agreed to by stakeholders to support operational risk decisions.

5.2 Protect

  1. (a)

    Strengthen the Access Control policy and procedures for organisation-wide assets that require both physical and remote access.

  2. (b)

    Sensitise and increase awareness about cyber security throughout the workforce more comprehensively and provide adequate cyber security training based on roles and responsibilities. In this regard, clearly describe cyber security roles and responsibilities for relevant staff and external stakeholders.

  3. (c)

    Enforce required provisions for data security in the policy and implement data-at-rest and data-in-transit security, and integrity-checking mechanisms to ensure confidentiality, integrity, and availability of information and data.

  4. (d)

    Establish required policies, processes, and procedures to manage protection of information assets. This include establishment of lacking policies and processes, particularly for configuration management, data destruction, and physical operating environment; identification of security baselines; SDLC for system management; formulate vulnerability, response, and recovery plans.

  5. (e)

    Strengthen processes that control and log remote access to organisational assets by external maintenance contractors.

  6. (f)

    Establish a central log of organisation-wide information systems and devices, establish Removable Media policy, and strengthen network segregation to protect communications and controls networks.

5.3 Detect

  1. (a)

    Determine baselines for network operations and data flows, implement appropriate activities to detect and analyse events based on event data aggregated from multiple sources and sensors. Determine incident impact and threshold to prepare and allocate resources appropriately.

  2. (b)

    Implement tools to monitor cyber and physical environments to detect unauthorised mobile code, external service provider activities, and unauthorised access. Perform organisation-wide vulnerabilities regularly.

  3. (c)

    Outline detection requirements in Information Security policy and continuously improve these processes to ensure timely and adequate awareness of anomalous events.

5.4 Respond

  1. (a)

    Establish processes and procedures to respond to cyber security events in a timely manner.

  2. (b)

    Define cyber security roles and responsibilities in Information Security policy to ensure activities are coordinated for internal and external stakeholders including law enforcement in response to cyber security events.

  3. (c)

    Implement required cyber security events notification and detection systems to ensure adequate information is available to analyse and understand the impact to support recovery activities.

  4. (d)

    Implement required cyber security controls to detect, report, and contain incidents to prevent escalation of an incident, mitigate its effect, and eradicate the incidents.

Each of the above recommendations also had specific internal stakeholder(s) identified to indicate ownership and responsibility for addressing the issues associated. Consequently, the organisation was then able to develop strategies to address the issues identified, and assign specific tasks to individuals. For this purpose, the organisation established an internal document using Microsoft Power BI [25] (typically referred to as a Power BI site) to track and visualise the status of the NIST CSF assessment (Fig. 1).

Fig. 1
figure 1

Microsoft Power BI Internal Site for tracking, visualising, and reporting NIST CSF assessment findings, courtesy of the participating local government organisation

Full size image

The Power BI site facilitated transparency, visibility, and central reporting throughout the organisation. Intuitively, this resulted in a rapid and responsive drive for the organisation to address and prioritise issues based on severity and cost, with the goal of achieving Tier-2 compliance.

Furthermore, a desire to achieve a higher compliance level such as Tier-3 was expressed. Such aspiration is encouraged, however, with caution. Even though a higher level of compliance will improve the cyber security posture of the organisation, it will also affect other aspects such as resources and cost. For example, when contrasting the Risk Management Process between Tier-2 and Tier-3 as defined in the NIST CSF [28]:

  1. (a)

    Implementation of risk management practices are not mandatory in Tier-2, whereas these have to be implemented as organisation-wide policies in Tier-3. Thus, Tier-3 organisations should have procedures, processes, technology, and human resources to implement relevant policies.

  2. (b)

    The cyber security activities’ priorities are updated in a passive nature in Tier-2 as opposed to regular active updates and constant re-evaluation of priorities for Tier-3 compliance. To acquire such capability, an organisation requires adequate technology, skilled human resources, and relevant policies that would enable keeping pace with the changes in the technology and threat landscape.

In addition to the two points highlighted above, considering both Integrated Risk Management Program and External Participation [28], significant investment in resources and human skills development or acquisition is needed to make the transition from Tier-2 to Tier-3. Moreover, this should only be considered carefully based on the organisation’s business requirements, strategic objectives, budget, risk appetite, and current and future threats.

Table 2 Summary of case studies for NIST SP 800-53
Full size table
Table 3 Summary of case studies for ISO 27001:2013
Full size table

6 Related frameworks

The diversity and complexity of Information Technology (IT) system components have increased significantly in recent years. Consequently, in order for businesses to adequately secure these systems, several standards and frameworks have been developed [2]. Such frameworks need to be applicable to all manner of business sectors, be they government or private, enterprise or small-business. Tables 2 and 3 provide a summary of useful examples of how both NIST SP 800-53 and ISO/IEC 27001:2013 frameworks have been applied in practice.

Since NIST CSF can be considered as a high-level abstraction of related frameworks, it provides references to other related frameworks for specific implementation guidelines. These referenced frameworks include:

  • NIST SP 800-53 Rev. 4.

  • Control Objectives for Information and Related Technologies (COBIT5).

  • ISO/IEC 27001:2013.

  • ISA 62443-2-1:2009.

  • ISA 62443-3-3:2013.

These are further described below.

6.1 NIST SP 800-53 Rev. 4

NIST SP 800-53 [27] revisions are made according to changes in responsibilities under the Federal Information Security Management Act (FISMA), Public Law (P.L.) 107-347. The latest version of this framework consists of five functions (Identify, Protect, Detect, Respond, and Recover), 22 categories, and 98 subcategories. This framework utilises a four-tier security model (Partial, Risk Informed, Repeatable, and Adaptive) and a seven-step process (Prioritise and Scope, Orient, Create a Current Profile, Conduct a Risk Assessment, Create a Target Profile, Determine, Analyse and Prioritise Gaps, and Implement Action Plan). It focuses on assessing the current situation by determining how to assess security, how to consider risk, and how to resolve the security threats.

6.2 Control Objectives for Information and Related Technologies (COBIT5)

COBIT5 [17] is a business CSF designed for the governance and maintenance of enterprise IT systems. It consists of five domains and 37 processes in line with the responsibility areas of plan, build, run, and monitor. COBIT5 is aligned and coordinated with other recognised IT standards and good practices, such as NIST, ISO 27000, COSO, ITIL, BiSL, CMMI, TOGAF and PMBOK. It is built around the following considerations:

  • The need to meet stakeholder expectations.

  • The end-to-end process control of the enterprise.

  • To work as a single integrated framework.

  • Recognising that “Management” and “Governance” are two different things.

6.3 ISO/IEC 27001:2013

ISO/IEC 27001:2013 [23] is an international information security standard published by the International Organisation for Standardisation (ISO) and the International Electrotechnical Commission (IEC), which originated from the British Standard, BS 7799. This framework consists of 114 controls in 14 groups describing the requirements needed to design and implement an Information Security Management Systems (ISMS). Version 2 released in 2013 replaces the 2005 version 1 edition. It is a standard that should be instigated by all businesses where information security is a critical factor, but in particular, applies to software development, managed service providers/hosting services providers, IT, banking and insurance, information management, government agencies and their service providers, and E-commerce merchants [23].

6.4 ISA 62443-2-1:2009

ISA 62443-2-1:2009 [21] is an International Standards on Auditing (ISA) standard covering the elements required to develop an Industrial Automation Control System Security Management System (IACS-SMS). It consists of three categories, three element groups, and 22 elements. The framework is the first of four ISA policy and procedure products that identifies the essentials necessary to establish an effective cyber security management system (CSMS). However, the step-by-step approach as to how this is achieved is company-specific and according to their own business culture. These essentials are:

  • Risk analysis.

  • Addressing risk with the CSMS.

  • Monitoring and improving the CSMS.

6.5 ISA 62443-3-3:2013

ISA 62443-3-3:2013 [22] is an International Standards on Auditing (ISA) standard covering the elements required for cyber security controls of industrial control systems (ICS). It consists of seven Foundation Requirements and 51 System Requirements.

ISA 62443-3-3:2013 is the third of three ISA systems products, that outlines system security requirements and security levels [22].

6.6 Other frameworks

In addition to the above, other frameworks used in the industry include:

  • Committee of Sponsoring Organizations of the Treadway Commission (COSO) is an enterprise risk management standard, designed jointly by five leading associations, with the aim of integrating strategy and performance [13].

  • Council on CyberSecurity Top 20 Critical Security Controls (CCS CSC) consists of a prioritised set of actions, originally developed by the SANS Institute, to protect assets from cyber attack [12].

  • ISF Standard of Good Practice (SoGP) is a standard aimed at providing controls and guidance on all aspects of information security [20].

  • ETSI Cyber Security Technical Committee (TC Cyber) was developed to improve standards within the European telecommunications sector [15].

  • Sherwood Applied Business Security Architecture (SABSA) Enhanced NIST Cybersecurity (SENC) project enhances the five core levels of the NIST CSF into a SABSA model consisting of a six-level security architecture [30].

  • IASME Consortium (IASME) is an information assurance standard based on ISO 27000, but aimed at small businesses [18].

  • RFC 2196 - Site Security Handbook (SSH) represents a guide on how to develop computer security policies and procedures [19].

  • Health Information Trust Alliance (HITRUST) is the first IT security CSF designed specifically for the healthcare sector. It is based on existing NIST standards and is aimed at healthcare and information security professionals [16].

  • North American Electric Reliability Corporation Critical Infrastructure Protection (NERC-CIP) version 5 is a set of requirements needed to secure the assets of the North American bulk electric system [14].

  • Open Security Architecture (OSA) is a free community-owned resource of advice on the selection, design, and integration of devices required to provide security and control of an IT network [29].

  • Good Practice Guide 13 (GPG13) is a UK government CSF related to Code of Connection (CoCo) compliance for businesses to secure IT systems [10].

7 Conclusion

In this paper we have used the NIST CSF to evaluate the cyber security risks of a local government organisation in Western Australia. Our approach can be used to derive measurable metrics for each Framework Core function and respective categories, thus enabling the organisation to ascertain the cyber security preparedness to actual risk.

Our findings suggest that evaluating the Desired Tier compliance to the NIST CSF helps identify the specific people, process, and technology areas that require improvement (i.e. gaps), which directly influence threat mitigation. The application of CSF helped us understand the current security context of the organisation while identifying the risks and future growth areas to improve. While higher tier compliance maybe desired, we have also recommended that the organisation’s business requirements, strategic goals, budget, risk appetite, and current and future threats to be considered carefully.

Furthermore, as we have presented several related frameworks, navigating such frameworks for self assessment can be challenging, often not intended by design even, but not impossible. We have observed that the NIST CSF offers an advantage over other frameworks in this regard. However, there is still room for developing additional tools that would simplify the implementation process and speed up adoption.

Therefore, our future work will aim to improve the current Assessment Tool we have used, with a focus of making it adaptable and accessible to a wider audience and measurable for accurate quantification of cyber preparedness.