Probabilistic dynamic programming algorithm: a solution for optimal maintenance policy for power cables

This paper presents a probabilistic dynamic programming algorithm to obtain the optimal cost-effective maintenance policy for a power cable. The algorithm determines the states which a cable might visit in the future and solves the functional equations of probabilistic dynamic programming by backward induction process. The optimisation model considers the probabilistic nature of cables failures. This work specifies the data needs, and presents a procedure to utilize maintenance data, failure data, cost data, and condition monitoring or diagnostic test data. The model can be used by power utility managers and regulators to assess the financial risk and schedule maintenance.


Introduction
Power cables play an integral part in the transmission and distribution of electricity. The reliability of power cable contributes substantially towards the reliability of the entire electrical distribution network. The unexpected outages due to the failure of the power cables have a severe impact on utility companies due to tight economic requisites and regulatory pressure. This has engendered a demand for high reliability and a need for the extension of cable life with minimum maintenance cost which can only be achieved by implementation of an effective maintenance policy.
In recent years, many methods have been proposed and utilized for the maintenance and replacement of engineering assets; among them, dynamic programming is the most widely used. The dynamic programming approach can provide the optimal cost-effective and reliability-centered maintenance policy for the assets which are required to operate indefinitely. Moghaddam and Usher (2011) presented two dynamic programming-based models to determine the optimal maintenance schedule for a repairable component which has an increasing failure rate. The objective of the two models was to obtain maintenance decision, such that it minimizes total cost subjected to a constraint on reliability and maximizes reliability subjected to a budget constraint on overall cost. In another paper, Korpijärvi and Kortelainen (2009) showed the application of dynamic programming for the maintenance of electric distribution system. Abbasi et al. (2009) developed a priority-based dynamic programming model to schedule the maintenance of the overhead distributed network. They adopted a risk management approach to consider the actual condition of the electrical components and expected financial risk in the model. An application of dynamic programming for maintenance of power cable was presented by Bloom et al. (2006). The model represents life-cycle cost approach and it can provide an appropriate time to utilize diagnostic test information in a cost-effective manner. However, the model fails to consider the random failure behaviour of the cable and does not optimize the cost of different maintenance decisions.
A large number of reliability centered maintenance (RCM) optimization methods are presented for electrical power distribution system. Recently, multi-objective genetic algorithm to minimize preventive maintenance cost while maximizing the reliability index of the whole system was presented by Piasson et al. (2016). This method optimizes only PM cost and reliability index does not consider the ageing of cable insulation. Yassad et al. presented two system-level RCM optimization methods (Yssaad et al. 2014;Yssaad and Abene 2015). Both 1 3 methods identify the components which require special attention and its goal is to minimize the corrective and preventive maintenance cost by maximizing reliability. The modeling technique was based on functional and dysfunctional failure analysis of failure modes using the FMEA model (Yssaad and Abene 2015). These methods do not consider all maintenance decision-preventive maintenance, corrective maintenance, and replacement. They have not explored the rationale behind length planning horizon and failed to consider expected lifetime of the components and impact of maintenance.
In this paper, probabilistic dynamic programming algorithm is proposed to obtain optimal cost-effective maintenance policy for power cables in each stage (or year) of the planning period. In this model, the length of the planning horizon is equivalent to the expected lifetime of the cable. The expected life of the cable is obtained from the previously developed ageing model based on stochastic electro-thermal degradation accumulation model. The maintenance policy in this model includes preventive maintenance, corrective maintenance, replacement, and do nothing as a set of decisions. The algorithm first finds the future state of the cable by qualifying the effect of each maintenance decision, and then, it uses backward induction method to solve the dynamic programming recursive equations consisting of future state transition probability. The random failure behaviour of the power cable is included in the model by considering it as a stochastic or random process. This work specifies the process of applying the failure data, maintenance data, and diagnostic test data in the decision-making process. The proposed methodology can also be used in the maintenance of other electrical components, as well.

Proposed methodology
The proposed methodology for estimation of optimal maintenance policy for each stage of the maintenance period is shown in Fig. 1. The algorithm has two parts. The first part of the algorithm is utilized to obtain all possible states which a cable might visit in future by quantifying the effect of maintenance actions on cable state. In the second algorithm, future state from the first algorithm, transition probabilities of future state, and maintenance costs are utilized as an input in the model to calculate the optimal maintenance policy by solving the recursive equations.

Probabilistic dynamic programming algorithm
3.1 PART 1: estimation of future state of the cable

Length of maintenance period
Length of planning horizon could be finite or infinite. The infinite planning horizon is often assumed when it is difficult to establish a termination time. At the same time, an inappropriate choice of finite planning horizon affects the validity of the model. The power cables can operate a certain number of years before they become completely obsolete. A cable has two types of failure criteria. First criteria are focused on the decline in the performance of cable insulation and second criteria are focused on the loss of ability to resist fire (Yang et al. 2016). A cable has a finite lifetime. Therefore, it is very important to establish a rationale for the end of the cable lifetime (Mazzanti 2007). The lifetime of the cables is usually obtained by modeling the historical failure data which have high fluctuations due to the presence of both random and ageing failures (Sachan et al. 2015a, b). The fluctuating data source should not be utilized to develop the long-term maintenance policy which includes proactive replacement as one of the high investment maintenance decisions. Cost of corrective or preventive failure is much less than completes replacement. The corrective maintenance restores the cable back to its operational state after the occurrence of a failure by cutting and splicing in a new cable section. The preventive maintenance improves the reliability by detecting the potential failures.
Power cable failure occurs due to random, ageing, or cumulated effect of both the causes. A random failure can occur due to degradation in a small section of a cable circuit such as poor workmanship, a manufacturing defect, or sudden mechanical (Sachan et al. 2015b), whereas ageing failures occur in cable insulation due to dominant electrothermal stress in daily load cycle (Sachan et al. 2015a, b). Most common mode of insulation failure is electrical breakdown of insulation, breakdown at the electrical interface, and insulation thermal breakdown (Dong et al. 2014;Orton 2013). The insulation is the weakest link of a power cable in terms of degradation or failure. Completely  degraded insulation leads to unrecoverable failure; after this type of failure event, any kind of maintenance action is ineffective. Therefore, it can be hypothesized that the life of a cable is equivalent to the time to degradation of the cable insulation (Mazzanti 2007;Sachan et al. 2015a). Throughout the world, power distribution networks have high concentration of polymeric-insulated cables. Crosslinked polyethylene (XLPE), ethylene propylene rubber (EPR), and their superior versions such as tree-retardant cross-linked polyethylene (TR-XLPE) are used to insulate the conductor of the cable. In this research, a finite planning horizon for the maintenance of power cables is determined by a previously developed stochastic electrothermal model to estimate the residual life of the cable based on degradation of polymeric insulation (Sachan et al. 2015a).
The model provides accumulated degradation level by considering seasonal load cycle, conductor temperature, and seasonal soil or atmospheric temperature. The degradation can be quantified in terms of percentage with the advancement of age for a group of cable with similar installation year, design, and operational conditions. The degradation level and planning horizon of cable population installed in year i 0 and i 1 is shown in Fig. 2.

Set of states and maintenance decisions for each stage
Let the maintenance period starts from y = 0 to y = Y , and the time unit for y could be in months or yearly, as a decision of maintenance can be taken monthly to yearly basis. Each time unit y of maintenance period is called stage. At any stage y of the maintenance period, a cable can either be in an operating state with effective age a ′ y or in failed state F a ′ y . F a ′ y is the failure at an effective age a ′ at stage y.
Four types of maintenance decisions are taken on a cable asset: "no action" NA, "preventive maintenance" PM, "replacement" RP, and "corrective maintenance" CM. Here, NA means take no maintenance action on cables. The preventive maintenance is taken to reduce potential failures in near future. The corrective maintenance is only carried out on cables in a failed state. The replacement action renews an old cable with a new cable.
A maintenance decision is taken at beginning of each stage y . The maintenance decision depends on the state. At operating state (a � y ) , NA, PM, and RP decisions are taken for The decision of corrective maintenance is not take at the final stage ( y = Y ) of planning horizon. Only, RP decision is taken in failed (F a � Y ) or operating state (a � y ) , at the final stage of planning period y = Y , when a cable fails at the end of its lifetime and maintenance after this stage may not have any effect on the cable. Maintenance decision for failed and operating states of cable at different planning period is shown in Table 1.

Effect of maintenance
Maintenance has a positive and, sometimes, negative impact on an asset. The risk of failure of an important asset like cable can translate into the financial burden for both utilities and customers. Risk can never be eliminated completely, though the probability of occurrence of unwanted events can be reduced by planning effective maintenance practices. Effect of maintenance on the cable must be quantified appropriately to make an effective maintenance plan.
Risk of cable failure can be quantified by the probability of failure which changes with the advancement of service time (age) of a cable. The probability of failure is estimated from either time-to-failure data or failure count. The time-to-failure data can be modeled by the Weibull distribution. Failure events in Weibull distribution are assumed to be independent and identically distributed (i.i.d). It treats cable as a non-repairable component or it was not maintained in the past (Tang et al. 2015). Non-homogenous poisson process (NHPP) is also NA, PM, RP CM RP utilized to model both time-to-failure and failure count data. The failure events in NHPP models are not independent and identically distributed. Thus, it considers the fact that cable is a repairable component (Sachan et al. 2015b). In this research, cable is assumed to a repairable component. In the numerical example, as shown in Sect. 4, the failure probability of cables was obtained by NHPP. A detailed application of NHPP on power cable can be seen in Sachan et al. (2015b). Let, failure distribution of cables homogenous in terms of voltage level, insulation material and installation year is where y is the failure time (age at which cable failed) and a is the age of the cables. The probability of failure of cables under no maintenance or unidentified past maintenance practices is shown in Fig. 3. Maintenance activity such as preventive maintenance (PM) action reduces the failure probability; however, the PM methods can only detect some potential failure causes and other causes remain undetected. Contribution of PM methods towards the reduction in failure probability of cable can be obtained by Eq. (2). It can be assumed that the failure probability reduces by the same percentage and this affects the relative age of cable in comparison to cables without maintenance (Bertling et al. 2005). Let, PM method z , z in {0, … , ℤ} can reduce failure probability by PM z % . The total reduced probability of failure is as follows: where a in {0, … , A} and a ′ in 0, … , A � is chronological age and effective age, respectively. Maintenance planning starts from y = 0 ; at this stage, the chronological age of cable is a . Before y = 0, information regarding maintenance on this cable may or may not be available. However, it is assumed that failure data are available, from which failure probability is obtained (by NHPP) and is predicted beyond y = 0 to show failure probability of cable if it is not maintained beyond this stage, as shown in Fig. 3. The probability of failure increases with time. Time is y the planning stage and chronological age a . Maintenance activities decrease the probability of failure and it extends useful life of the cable. The effective age after maintenance is used to reflect the impact of maintenance. Table 2 shows the impact of maintenance by effective age. If the probability of failure of a cable after maintenance is less than just before maintenance, then maintenance has a positive impact on the condition of the cable and its effective age is less than chronological age, a ′ < a. Similarly, if the failure probabilities remain same, then maintenance has no effect on cable condition and effective age is equal to chronological age, a � = a . A failed cable is repaired by corrective maintenance (CM). It could restore a cable to "good as new" state with effective age equal to 1, "worse than before" state a ′ > a , and "bad as old" state,

Transition probability
A maintenance decision () on a cable at any stage y transforms it to another state at next stage y + 1 . Transition probability depends on current state and maintenance decisions  = {NA, PM, CM,RP} . By taking these decisions, a cable may transit either to operating state or failed state at stage y + 1 from its previous states at stage y . The probability of

(B) Preventive maintenance (PM)
The PM decision at state a ′ y can detect PM% of failures and reduce the failure probability by the same percentage. The undetected failure causes and a few unsuccessful PM actions eventually transit cable to the failure state in next stage y + 1 , as shown in Fig. 4.
The transition probability for PM action is as follows: The RP action on cable at stage y results in age 1 at next stage y + 1 . The power cable has a life longer than 20 years. A study has shown the cable life scenario (Sutton 2011). According to this study, XLPE, TR-XLPE, and EPR cables have a lifespan of 30, 50, and 45 years, respectively. Manufacturing techniques, material, design, and installation method improve within a few years of time frame (Orton 2013(Orton , 2015. At the same time, maintenance practices and techniques are to detect faults in cable changes, as well.
At stage y , new cable is replaced by old cable. At next stage y + 1 , new cable will have age 1. Successful transition of new cable to the next stage is highly dependent on the quality of cable and installation practices. Manufacturers conduct quality tests on each cable section to detect the expected fault. Transportation of cable to site and installation activities can cause damage to the cable (Dong et al. 2014). Transition probability of new cable to next stage can be estimated by infant mortality rate of those cables. For simplicity, it can be assumed that installation practices are reasonability accurate, failure probability is negligible (0.01) at age 1 due to very low infant mortality rate and it would be highly likely (0.99) that cable will transit to an operating state a � y+1 = 1 , shown by the following equation: (D) Corrective maintenance CM decision is taken when a cable is in failed state F a ′ y . Cable can regain its operating state ( F ) or it can again land to a failed state ( F ) after repair by corrective maintenance. The cable repair during CM could be perfect, minimal, and worst. CM would restore cable to an operating state with "good as new", "bad as old", "worse than before", and failed conditions. This can be established by studying the past maintenance data. For simplicity, it can be assumed that the CM restores cable to a condition to "bad as old" (neutral state, as shown in Table 2) with F CM probability. It means that repair action will bring a cable back to its operating state; however, maintenance would have neither positive nor negative effect. This implies that, effective age equal to its chronological age at stage y + 1. The transition probability for CM activity can be estimated by available maintenance record data:  Figure 5 shows the transition in the future state. At any stage (y) , a cable can either be in operating state or failed state, from there, it can transit to future states with F  and F  probability of transition to failed and operating state at stage (y + 1) , respectively, as a result of carrying out maintenance decisions  = {NA, PM, RP, and CM}.

Maintenance cost
The optimal decision policy depends on four types of cost: A. Replacement cost The cost of replacement (C RP ) of a power cable in a distribution network is as follows: In Eq. (7), C cable is the cost of cable per km ; C inst is the installation cost per km which includes service charges of engineer, cost of dismantling, decommissioning, and transportation, and l is the length in km.

B. Failure and unplanned interruption cost
The underground power cables have four types of interruptions: unplanned, planned, high-speed auto-reclosing (AR), and delayed AR (Lassila et al. 2005). The cost of failure due to unplanned outages in a network depends on the customer group. For example, the unit cost of failure ( C∕kW ) is higher in industrial, public and service sector customers than the residential and agricultural customers. The cost of an unplanned outage or failure (C F ) for customer group is (Lassila et al. 2005): In Eq. (8), d is outage cost for the power not supplied to the customer group ($∕kW) ; b is the time-dependent power outage cost for the energy not supplied to the customer group ($∕kWh) ; t r is the average unplanned interruption time (h) and L is the average hourly power consumption of customer group . (7)

C. Maintenance cost
The maintenance of underground cables alleviates many potential failures. The implementation of maintenance activity depends on the past failure causes. The external failure modes, change of soil condition, and level of water or moisture can be detected by routine visual inspections, and the other obvious failure symptoms can be detected and prevented by the diagnostic tests such as partial discharge detection (Lassila et al. 2005). The annual maintenance cost per km is (C PM ): In Eq. (9), C m is the preventive maintenance (PM) cost of any method m with m = 1 to  . The PM methods could be silicon injection rehabilitation, inspection, and diagnostic tests.

D. Repair cost
The cost of repairing a failed cable consists of fault location cost and the cost of repairing a fault. The cost of detecting the exact fault location in an underground cable is much higher than overhead cable. The current preventive maintenance practice and technology is not capable of detecting all the failure causes. Therefore, there are two possible kinds of repair. First, repair when the potential failure causes are detected by PM . Second, repair when corrective maintenance ( CM ) is carried out on failed cable when the failure cause remains undetected and eventually fails in the future. The CM repair cost (C RE CM ) is given by the following: In Eq. (10), C f _det is the cost of fault detection per km , l is the length in km , and C AR is the average cost of fault repair. The PM repair cost C RE PM is usually less than CM repair cost (C RE CM ) , because CM repair action is taken after the occurrence of the failure which includes a high cost for detection and repair of a failed section of the cable. The PM repair cost depends on the type of preventive maintenance action taken on the detected potential failure location. For (10) C RE CM = C f _det l + C AR . example, silicon injection rehabilitation is one of the effective methods to prevent water tree in the early produced (the 1970s) XLPE cables (Ma et al. 2016).

Objective and recursion equation formulation
Objective Obtain optimal maintenance policy that minimizes the total maintenance cost over a finite planning horizon 0 < y < Y . The total cost of no action on a cable, replacement, preventive maintenance, and corrective maintenance decision is given by the following equation: The objective is achieved by solving bellman equations by backward induction for all the possible states which a system might visit in future (Sachan et al. 2015c). The basic structure of bellman equation is as follows: The backward induction process proceeds by first finding the minimum maintenance cost for all states at the last stage (12) Current cost = immediate cost + future cost.
cost; preventive maintenance (PM) has an immediate cost of maintenance and repair, replacement (RP) has an immediate cost of replacement, and corrective maintenance (CM) has an immediate cost of failure and repair: for y = Y, The cost of maintenance decisions at effective age (a � ) and fail ( F) state for stage y = 0 to Y − 1 is shown in Eqs. (16) and (17). The no action decision (NA) has no immediate V Y (F) = min(RP: C F + C RP ) = C F + C RP .

Numerical example
A numerical example is presented on an XLPE cable to illustrate the model. The cable was installed in the year 1984. It is a lateral cable which distributes electricity to a residential area of 34 houses. The probability of failure and XLPE insulation degradation level is shown in Fig. 6. The methodology to estimate the failure probability by stochastic point process model based on the non-homogenous Poisson process and information about these cables is shown in Sachan et al. (2015a). In addition, the methodology to estimate the insulation degradation level based on the non-stationary Gaussian process is shown in Sachan et al. (2015a, b). These cables have an increasing failure rate as they suffer from a large number of random failures, especially due to water treeing as of lack of protective jacket.
In this example, the year 2016 is considered as the current year and optimal maintenance plan is launched from this year. The chronological age of cable at 2016 would be (17) V y (F) = min CM: . a = a , = 33 . At the initial stage y = 0 , the effective age is equal to chronological age. It was estimated that, by the years 2030 and 2055, the entire insulation of the cable is expected to reach 75% of moderately severe and 99.8% of severe of insulation degrade level, respectively (can be seen in Fig. 6). The optimal maintenance policy was found for two maintenance time period to show the outcome of the model for time period before the end of life and until the end of expected lifetime. The optimal cost-effective maintenance policy was found for two maintenance periods, first from the years 2016-2030 (stage ∶ y = 0 to 14) and second from the years 2016-2055 (stage ∶ y = 0 to 39)..
The first part of the algorithm shown in "Appendix A" was utilized to estimate the future state of the cable, as shown in Fig. 7. In future, decisions NA, PM, CM, and RP lead cable to an operating state with effective age a , and failed state F. The blue arrow in Fig. 6 depicts the chance of reaching failed state due to unsuccessful attempt of maintenance.
The second part of the algorithm computes the bellman equations by backward induction, i.e., from y = Y to y = 0 . It searches for optimal maintenance policy by visiting all the future states of each stage y (Bertling et al. 2005). The input maintenance and failure cost are shown in Table 3. The costs assumed in the model was populated by studying the economic analysis of cost parameters in Bertling et al. (2005), Lassila et al. (2005), and Ma et al. (2016). It should be noted that, usually, the cost of preventive maintenance is low in comparison to repair, replacement, and failure cost. The failure cost of a cable depends on the consumption profile of the customers which has a huge impact on the result of the model. The failure cost is low in this case, because the lateral cable serves residential customers. The probability of transition for no action (NA) from failure distribution, corrective maintenance (CM) and replacement (RP) decisions is shown in Sect. (3.2.1). Transition probability of preventive maintenance PM decision is obtained by assuming that only 60% (0.60) of potential failure causes can be detected (DET) and rest 40% (0.40) remain undetected (U_DET), and there is 0.98 and 0.02 chance that PM action would be successful and unsuccessful, respectively. The transition probability of PM action is F PM = 0.59 and F PM = 0.41 [from Eq. (4) F PM = 0.60 × 0.98 and F PM = 0.40 + 0.60 × 0.02]. The failure probability of 0.08 (8%) is assumed as the minimum acceptable level. The PM and RP decisions are not taken below this level.
The algorithm suggests the PM at y = 1 , y = 8 and replacement ( RP ) at y = 18 (2034) as the optimal decision policy for lengthiest planning horizon y = 0 to 39 . The result shows that the application of PM can retain the cable in service till y = 14(2030) with minimum maintenance cost at moderately severe insulation condition. However, the cable must be replaced with a new XLPE at or just before y = 18 (2034), because, at this year, the cable maintenance cost exceeds replacement cost and entire insulation is expected to have severe degradation. The severe degradation in entire insulation and high maintenance cost compared to replacement cost is justifiable a reason to support the proactive replacement of the unjacketed cables between the years 2030-2034 ( y = 14 -18 ). The optimal maintenance policy for both time periods is shown in Fig. 8.

Conclusion
The proposed probabilistic dynamic programming model is capable of finding the optimal decision policy with respect to optimal long-run cost for a cable with a known failure distribution and degradation level. The optimal policy improves the reliability by suggesting the appropriate time for preventive maintenance and replacement action. The utilities and regulators can assess the monetary risks by exploiting the probabilistic nature of the model.
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creat iveco mmons .org/licen ses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

A1: Data structures
A. Vector ST (y) It stores effective age (a � ) and failed (F) states of a cable for each planning stage y = 0 to Y. The algorithm initializes the current effective age at y = 0 and stores it at ST 0 vector. It determines the futures states in first part of the algorithm. B. Matrix R y For each planning stage y, the result is stored in matrixes which have two columns and rows equal to the number of expected states at any stage y of the planning horizon. The first column of the matrix stores state of the cable and second column matrix stores minimum cost for maintenance action for a given state.
The algorithm has two parts. The first part finds the future states when current effective age is known. Please note that   stage y = 0 is the current stage, where the effective age is equal to the current chronological age a � = a . In second part, for each stage, the algorithm finds the minimum cost of a maintenance action for all the cable states. The algorithm solves the problem by computing backwards towards the initial time. It finds the minimum cost for y = Y , then y = Y − 1, then y = Y − 2 and so on. The Algorithm is presented in Table 4.