Grid Emergency Decision Technology

Abstract

From the perspective of system theory, the so-called mechanism refers to the operation mechanism and working principle of the system, and in fact, it is the operation mode of the internal components of the system and the operation rules and principles of the mutual connection and interaction between the elements.

6.1 Power Disaster Contingency Posture Map

6.1.1 Mechanisms for the Evolution of Situational Emergencies

From the perspective of system theory, the so-called mechanism refers to the operation mechanism and working principle of the system, and in fact, it is the operation mode of the internal components of the system and the operation rules and principles of the mutual connection and interaction between the elements. As a complete system, emergencies also have their own evolutionary mechanism. The evolution mechanism of emergency events can be divided into two aspects: occurrence mechanism and development evolution mechanism, among which the development evolution mechanism can be divided into four aspects: spread, derivative, transformation and coupling.

• (1) Mechanism of emergencies

Emergencies generally occur under the combined action of intrinsic causes, direct causes and indirect causes in a specific time, space and external environment. The essential cause of emergencies is composed of the self-defects of people, things and the environment, of which human defects include human physical defects and consciousness defects, material defects include defects in the self-aspect of things and defects in the state of things, and environmental defects include natural environment defects and socio-economic environmental defects; The direct cause of the emergency is composed of unsafe factors of people, things and the environment, among which human insecurity factors refer to people's unsafe behavior, material insecurity factors refer to the unsafe state of things, and environmental insecurity factors mainly refer to the negative impact of the environment; The indirect causes of emergencies are mainly caused by inadequate daily management, including inadequate personnel management, inadequate material management and inadequate environmental management, as shown in Fig. 6.1.

Fig. 6.1

Mechanism of sudden incident occurrence

The mechanism of emergency can be divided into two types according to the continuity of the change of the event itself: one is caused by qualitative change to a certain extent. This change is more of a continuous and gradual change process; the other is caused by mutation, which is caused by the sudden change of some factors or the external environment, which is more of a discontinuous and abrupt process. In reality, most of the emergencies are combinations of the above-mentioned two types, including not only the process from quantitative change to qualitative change, but also the sudden changes. In fact, regardless of the type of occurrence mechanism, it is due to the defects of the thing or the environment, which leads to the long-term existence of some potential risks. These potential risks suddenly break out in a specific time, space, environment and inducements, leading to the occurrence of emergencies.

• (2) The development and evolution mechanism of emergencies

After their occurrence, emergencies will evolve under the action of internal and external factors, leading to the development of emergencies in different directions, and some secondary disasters or derivative accidents may develop. Therefore, in order to timely and effectively deal with emergencies, make the development and evolution of the events develop in the expected direction, and minimize various losses, it is necessary to understand the development and evolution mechanism of the emergencies. Generally speaking, the development and evolution mechanism of emergencies can be specifically divided into four aspects: spread, derivation, transformation, and coupling, as shown in Fig. 6.2.

Fig. 6.2

Evolution mechanism of sudden incident development

• 1) Spread mechanism

The so-called emergency spread refers to the occurrence of an emergency, leading to more incidents of the same kind. This “more” can be spatial expansion, such as fire in power equipment, or time transmission or extension, such as debris flow, which may cause spread and damage to other power facilities over time. The spread mechanism is more of an expansion of the state of affairs, and the initial emergency will not die out because of the occurrence of subsequent events, but exist at the same time. Emergency contagion can also be the spread of online virtual environments, such as the spread of online rumors. Due to the rapid propagation speed, the wide-range influence, and large number of contacts, the spread on the internet is more harmful since it has rapid propagation speed, wide-range influence, and large number of contacts on the Internet, as shown in Fig. 6.3.

Fig. 6.3

Schematic diagram of the emergency network spread

• 2) Derivative mechanism

The so-called emergency derivation refers to the situation where the occurrence of an emergency event results in other types of more serious events due to excessive, harmful or improper response measures. For example, in order to cope with the traffic congestion caused by heavy snow, snow melting agents are sprinkled in a large area, leading to environmental pollution; Oil disinfectants to eliminate oil spills at sea may cause marine pollution; In 2011, the earthquake and tsunami in Japan destroyed the power system of the Fukushima nuclear power plant, and the destruction of the power system caused power outages.

Then the power outages make the cooling water unable to be replenished in time, and the drop in water level causes the nuclear fuel rod to be exposed to the surface of the water to meltdown, which eventually leads to a large amount of radioactive material leakage, resulting in a nuclear leakage accident. There are generally three development models for emergency derivation, as shown in Fig. 6.4 shown.

Fig. 6.4

Three patterns derived from the outbursts

• 3) Transformation mechanism

The so-called emergency transformation refers to the situation where emergencies are transformed into different types of new disaster events under the action of their own factors or external environmental factors (as shown in Fig. 6.5). The most important feature of the transformation mechanism is “replacing the old with the new”, that is, the occurrence of event A directly leads to the occurrence of event B, while event A disappears after event B occurs. For example, typhoon-induced rainstorm disasters are the embodiment of the mechanism of emergency transformation, and typhoon disasters are transformed into torrential rain disasters.

Fig. 6.5

Incident transformation mechanism

• 4) Coupling mechanism

The so-called emergency coupling refers to the phenomenon that two or more factors interact and influence each other, thus leading to the occurrence of emergencies or serious events. According to the characteristics of the object, it can be divided into event and event coupling, event and factor coupling, and factor and factor coupling. That is, when incident A induces incident B, C and D, incident A does not die, they interact with each other instead, as shown in Fig. 6.6.

Fig. 6.6

Mechanism of burst incident coupling

6.1.2 Analysis of the Evolution Patterns and Pathways of Emergencies

The situation of an emergency also evolves over time after its occurrence. Therefore, it is necessary to understand the scenario evolution law and evolution path in order to construct the scenario network model, predict the future development trend and influence factors of emergencies, so as to make coping strategies.

• (1) The evolution law of the emergency situation

The evolution law of emergency situation is the front and basis of analyzing the evolution path of emergency situation. However, due to the characteristics of being sudden, unpredictability, dynamics and serious consequences, the evolution process of emergencies is very complex and the law are difficult to find. In addition, in the whole process of emergency development and emergency response, the event scenario is also affected by external factors such as external environment and human intervention, which increases the complexity of the evolution of the whole event scenario. Although the evolution process of the emergency situation is complex, it does not mean that there is no regularity. After careful analysis, there is still a certain regularity. According to the scenario elements, the emergency response scenario can be divided into five element dimensions, namely the disaster-causing body, the disaster-bearing body, the external environment, the emergency resources and the emergency activities, of which the disaster-causing entity and the disaster-bearing body belong to the emergency object, the external environment and emergency resources belong to the external constraints, and the emergency activity belongs to the behavior and action of the emergency subject on the emergency object. If the emergency is regarded as a complete system, this division method mainly clarifies the components of the system from a static perspective.

The evolution of emergency scenarios is generally a dynamic process. Therefore, it is not only necessary to clarify the components of the scenario, but also more important to clarify the relationship and interaction between the components. In reality, emergency activities generally act on both the disaster and the carrier, the disaster and carrier can be combined together from a dynamic perspective to form the situational state elements, so that the complexity of the model can be reduced by reducing the number of elements. From this perspective, the scene evolution process of an emergency can be divided into the following four key elements: (1) scene state, represented by the letter S, mainly refers to the state of the emergency object, including the disaster state and the disaster object; (2) the emergency activity, represented by the letter B, refers to the disposal behavior and measures taken by the emergency subject to change the emergency state; (3) the external environment, represented by the letter H; (4) the emergency resources, represented by the letter M, which belongs to the external constraints of emergency activities. The interaction between these four key elements constitutes a basic unit, as shown in Fig. 6.7.

Fig. 6.7

Schematic diagram of the basic unit of the evolution of sudden incident scenarios

Since emergency resources do not directly act on the state of the scenario, emergency resources are considered as a constraint variable rather than an input variable in Fig. 6.7. In Fig. 6.7, S stands for the current situation. Under the influence of the external environment (H) and the intervention of the emergency activity of the emergency body (B), as well as under the constraints of emergency resources (M), the state of the situation changes and go into the next state (S₁), namely converting from the situation S to the situation S₁, realizing a complete process of situation evolution, called a basic unit of a sudden situation evolution.

Suppose an emergency undergoes situation conversion n times from its occurrence to its disappearance, the state of the situation is denoted as S₀, S₁, S₂, …, S_{n − 1}, S_n. S₀ is the initial state, and S_n is the state of disappearance. The time for each scenario is denoted as t₀, t₁, t₂, …, t_{n − 1}, t_n. B_i, H_i, and M_i respectively denote the emergency activities, the external environment and the constraints of emergency resource, i (1, 2, …, n). Then the evolution law of the emergency scenario can be expressed as shown in Fig. 6.8.

Fig. 6.8

The evolution of incident events

In Fig. 6.8, S₀ is the initial state of an emergency, the time of its occurrence is t₀. Under the intervention of emergency activity B₁, the influence of external environment H₁ and the constraint of emergency resource M₁, the state of the scenario changes, and enters the next state. Due to the different constraints of external environment and emergency resources, as well as different emergency activities, the evolution of the scenario is unpredictable, so the next state has multiple possibilities. Assuming at the time t₁, the state of the scenario is denoted as S₁, then under the intervention of B₂ and the influence of H₂ and M₂, the state of the scenario evolves again and a new state appears, and so on. Finally, at the time t_n, the situation disappears, the whole decision-making process of the emergency ends, and the evolution process of the scenario ends.

• (2) The evolution path of the emergency scenario

The so-called evolution path of emergency situation, in short, is the change trajectory of the emergency situation between the occurrence and the end of the situation, which also means expressing the current state of the emergency situation, the relationship between the situations, the evolution of the situation and the possible result of the evolution of the situation. By analyzing the evolution path of the scenario, we can accurately grasp the current state of the emergency situation, predict the development trend of the future situation, and make timely and correct decisions for the decision-making subject for practical reference.

After the occurrence of an emergency, if there is no intervention of the emergency subject, and it is only affected by the external natural environment, the evolution of the situation will generally develop and change according to its own laws and trajectory, showing the overall randomness and unique life periodicity. However, once the emergency occurs in reality, there will soon be an intervention of the emergency subject, so as to add human factors on the basis of the external environment. Therefore, the situational evolution should be influenced by human intervention, external environment and resource constraints, and its own evolution law and trajectory will be broken, presenting a new development and evolution law and evolution path.

Since the evolution of emergency scenarios is a continuous process, the emergency response is also a continuous process. The emergency decision-making subject makes emergency decisions and takes measures according to the state of the situation at the moment, so as to change the current state of the situation and enter the next state. Before making an emergency decision on the next state, the effect of last emergency decision-making is generally evaluated, and the results are generally divided into two types: meeting the expectations or failing to meet them. Meeting the expectation means that the emergency decision has the due effect, the situation has been controlled and developing in a good direction; Failing to meet the expectations means that the emergency decision is not effective or not obvious, and the situation is very serious or even worse. The decision-making subject should decide whether to make the next emergency decision before adjustment according to the evaluation results at any time. Because the results of each assessment are divided into two situations: meeting the expectations and failure, the evolution path can be divided into two directions: optimism and pessimism. Drawing on the principle of binary tree in computer data structure, the evolution path of emergency scenario can be formed as shown in Fig. 6.9.

Fig. 6.9

The evolution path diagram of sudden incident scenarios

In Fig. 6.9, there are 23 states of a scenario, where S₀ is the initial state after the occurrence, and S₁ to S₂₂ is the developmental evolution scenario. Each scenario state S_i (i from 0) under the action of emergency activity B_i + 1 and external environment H_i + 1, under the constraint of emergency resource M_i, the state changes to achieve scenario evolution. Each scenario evolution has two paths: the optimistic path (meeting the expectation) and the pessimistic path (failing to meet the expectation), where the horizontal right arrow (→) indicates meeting the expectation, the scenario evolution develops in the optimistic direction, and the vertical downward arrow (↓) indicates that the scenario evolution develops in the pessimistic direction. From this, Fig. 6.9 forms a most optimistic scenario evolution path: S₀ → S₂ → S₆ → S₁₄ → S₂₂, and a most pessimistic scenario evolution path: S₀ → S₁ → S₃ → S₇ → S₁₅. It can be seen from the above analysis that assuming after an emergency occurs, it totally undergoes situation evolution n times, that is to say, in addition to the initial situation, there are n states of situation, then theoretically speaking, there are 2n situation evolution paths. Each emergency decision determines the emergency in different directions, forming different evolution trajectory and evolution path, and there are only one most optimistic path and most pessimistic path. The most optimistic path is the best case, where every decision can meet the expectation; the most pessimistic path is the worst case, where every decision fails to meet the expectation. Therefore, due to the dynamic nature of the development of emergencies, the emergency decision-making subject should be very cautious when making every decision, and try their best to make the emergencies develop and evolve along the most optimistic path.

6.1.3 Construction of Electric Power Grid Disaster Emergency Situation Analysis Model and Scenario Simulation

According to the analysis of the emergency scenario evolution law and evolution path, it can be concluded that there are obvious causal link between different states of the situation. As the input variables, the current S_i, B_i + 1, H_i + 1 are the causes; As the output variables, S_i + 1 and B_i + 1 are the results; M_i + 1 are the constraints of causal conversion between input variables and output variables. Based on this, we plan to adopt the dynamic Bayesian network theory to construct the emergency scenario network.

• (1) Dynamic Bayesian network

Bayesian network (Bayesian Networks, BN), also known as reliability network, causal network or probability graph model, is an acyclic graph composed of representative variable nodes and directed edges connecting nodes (as shown in Fig. 6.10a). It is one of the most effective models in the field of uncertain knowledge expression and reasoning. The Bayesian network can be composed of three parts, expressed as: (X, A, P). X is the set of nodes in the network, and the elements in the set are random variables. A is the set of directed edges in the network, indicating the causal relationship between the nodes, expressed by x_i − x_j. x_i is called the parent node (the cause), x_j is called the child node (the result). P is the set of network parameters, and it is the probability value of the network node, expressed by p_i, and represents the conditional probability distribution of node x_i.

Fig. 6.10

Schematic diagram of Bayesian Network (a) and Dynamic Bayesian Network (b)

The mathematical basis of Bayesian network inference is the full and conditional probability formulas. If x is the set of causes or parent nodes of causality in a Bayesian network, y is the result or child node of causality in a Bayesian network, there is x → y, where the set x contains n elements and each element is denoted as x_i, x_i (i = 1, 2, …, n). The full probability formula is:

$$ \begin{aligned} P(y)\, = \, & P(yx) = P(yx_{1} + yx_{2} + \cdots + yx_{n} ) \\ \, = \, & P(yx_{1} ) + P(yx_{2} ) + \cdots + P(yx_{n} ) \\ \end{aligned} $$

(6.1)

As can be seen from Eq. (6.1), the full probability formula essentially pushes the result based on the cause. The posterior probability of the child node is calculated only given the prior probabilities and the conditional probability of the parent node to the child node. Another mathematical basis is the conditional probability, and the formula is as follows:

$$P(x\mid y)=\frac{P(xy)}{P(y)}$$

(6.2)

Taking Eq. (6.1) into Eq. (6.2) gives the following basic formula of the Bayesian network:

$$P({x}_{i}\mid y)=\frac{P({x}_{i}y)}{P(y)}=\frac{P({x}_{i})P(y\mid {x}_{i})}{{\sum }_{j=1}^{n}P({x}_{j})P(y\mid {x}_{j})}$$

(6.3)

As can be seen from Eq. (6.3), Bayes’ formula is essentially the opposite of the full-probability formula, which is to infer the probability of a cause when the result has already occurred.

Since Bayesian network inference all implies a former assumption of conditional independence, namely, for the set of parent nodes of a given node, this node is independent of all its non-descendant nodes. Therefore, the joint probability of all nodes represented by the Bayesian network can be expressed as the product of the conditional probabilities of each node, such as (6.4):

$$ \begin{aligned} P\left( {x_{1} ,x_{2} , \ldots ,x_{n} } \right)\, = \, & \prod\limits_{i = 1}^{n} {P\left( {x_{i} |x_{1} ,x_{2} , \ldots ,x_{i - 1} } \right)} \\ \, = \, & \prod\limits_{i = 1}^{n} {P\left( {x_{i} |P_{a} (x_{i} )} \right)} \\ \end{aligned} $$

(6.4)

P_a(x_i) is the parent node set of x_i.

The so-called dynamic Bayesian network (Dynamic Bayesian Networks, DBN) actually adds the time factor on the basis of the static Bayesian network, which makes the temporal reasoning and event development maintain the consistency and continuity in time, thus more in line with the objective reality. In essence, dynamic Bayesian networks can be regarded as an expansion of static Bayesian networks on the time axis (as shown in Fig. 6.10b). Assuming there are T time segments, n hidden nodes and m observation nodes, and x_ij represents the state of the i-th hidden node of the j-th time segment, it follows that:

$$\begin{array}{c}P({x}_{11},{x}_{12},\dots {x}_{T1},{x}_{T2},\dots {x}_{Tn}\mid {y}_{11},{y}_{12},\dots {y}_{1m},\dots {y}_{T1},{y}_{T2},\dots {y}_{Tm})=\\ \frac{P({x}_{11},{x}_{12}\dots {x}_{T1},{x}_{T2},\dots {x}_{Tn},{y}_{11},{y}_{12}\dots {y}_{1m},\dots {y}_{T1},{y}_{T2},\dots {y}_{Tm})}{\sum_{{x}_{11}{x}_{12},\dots {x}_{T1}{x}_{T2},\dots {x}_{Tn}}P({x}_{11},{x}_{12},\dots {x}_{T1},{x}_{T2},\dots {x}_{Tn},{y}_{11},{y}_{12},\dots {y}_{1m},\dots {y}_{T1},\dots {y}_{T2},\dots {y}_{Tm})}\end{array}$$

(6.5)

Since the dynamic Bayesian network itself also meets the conditional independence assumption, there are:

$$ P\left( \begin{gathered} x_{11} ,x_{12} , \ldots x_{T1} ,x_{T2} , \ldots x_{Tn} , \hfill \\ y_{11} ,y_{12} , \ldots y_{1m} , \ldots y_{T1} ,y_{T2} , \ldots y_{Tm} \hfill \\ \end{gathered} \right) = \prod\limits_{i,j} {P(y_{ij} |P_{a} (y_{ij} ))} \prod\limits_{i,k} {P(y_{ik} |P_{a} (y_{ik} ))} $$

(6.6)

After bringing formula (6.6) into formula (6.5):

$$\begin{array}{c}P({x}_{11},{x}_{12},\dots {x}_{T1},{x}_{T2},\dots {x}_{Tn}\mid {y}_{11},{y}_{12},\dots {y}_{1m},\dots {y}_{T1},{y}_{T2},\dots {y}_{Tm})=\\ \frac{{\Pi }_{ij}p({y}_{ij}\mid {P}_{a}({y}_{ij})){\Pi }_{i,k}p({y}_{ik}\mid {P}_{a}({y}_{ik}))}{\sum_{{x}_{11},{x}_{12},\dots {x}_{T1},{x}_{T2},\dots {x}_{Tn}}{\Pi }_{ij}p({y}_{ij}\mid {P}_{a}({y}_{ij})){\Pi }_{i,k}p({y}_{ik}\mid {P}_{a}({y}_{ik}))}\end{array}$$

(6.7)

$$i\in [1,T],j\in [1,m],k\in [1,n]$$

In Eq. (6.7), x_ij is a valued state, the first subscript i of each variable represents the time segment, the second subscript represents the j-th hidden node in the time segment; y_ij is the observation value, and P_a(y_ij) is the set of parent nodes of y_ij.

According to the basic principle and calculation process of the dynamic Bayesian network, it is necessary to construct the dynamic Bayesian network in three steps.

• 1) Determine the network node variable

A Bayesian network is composed of nodes. In order to build an emergency scenario Bayesian network, it is first necessary to select key elements according to the composition of the emergency scenario elements, so as to form node variables. After the node variable is determined, the value type and value domain of the node variable are also determined according to the actual situation.

• 2) Determine the relationship between the node variables

After the node variables are determined, the causal relationship between the node variables is determined according to the evolution law and path of the emergency scenario, and connected with directed line segments to form a directed acyclic graph network structure.

• 3) Allocation probability

After the Bayesian network structure is built, we need to assign probabilities to the nodes. There are two main types of node variables that need to allocate probability: one is the node variable without the parent node to determine the prior probability according to professional knowledge or experience, and the other is to determine the conditional probability for the node variable with the parent node.

The construction of a dynamic Bayesian network is a three-step process, and although it seems simple to describe, it is still very difficult in the actual operation process and requires a lot of professional knowledge. Therefore, for example, the determination of node variables, the description of causality and the allocation of probabilities require the assistance of experts in relevant professional fields. In addition, these three steps are not completed at once, and may require repeated refinement and correction.

• (2) Construction of the emergency situation analysis model

According to the dynamic Bayesian network construction steps of the emergency scenario mentioned above, it can be divided into the following three aspects.

• 1) Determination of network node variables

The determination of network node variables is the first and most important step in constructing a dynamic Bayesian network. The construction of emergency scenario dynamic Bayesian network is essentially a modeling process for the real world, and modeling is an abstract representation of the real-world process. Through modeling, the key elements that have an important influence on the objective system are abstracted, and the data of these key elements becomes variables. Therefore, determining the network node variables is actually abstracting the key elements of the objective system. The accuracy of the key elements directly determines whether the model can truly reflect the real system and the credibility of the final conclusion.

Due to the complexity and professionalism of the emergency scenario evolution, the determination of network node variables can be completed in steps: the first step is to collect the scenario elements according to the previous chapter; In the second step, the field experts will score the scenario elements and find out the key elements with the help of the technology of computer image recognition and matching. In the third step, based on the nature of the key elements, the types of the variables, namely, being continuous or discrete variables, are determined. Since the situational elements of emergencies are generally discrete, the transformed variables should also be discrete, which can be Boolean variables, order variables, or integer variables.

When an emergency occurs, we need to make decisions in a very short time in a tense environment, therefore, in order to avoid greater losses caused by decision-making errors, we generally adopt the pessimistic decision-making criterion, that is, considering more from the worst case. When the key elements are being selected through expert scoring, the score threshold should not be set too high to avoid resulting in the omission of some key scenario elements. The score threshold should even be set a little lower. Although some sub-key elements may be included to increase the total amount of elements, so as to increase the difficulty of modeling solution, it is better than missing some key elements, at least there will be no mistakes in matters of principle.

• 2) Determination of the causality of the node variables in the network

After the network node variables are determined, that is, the key elements of the model are determined, the next step is to determine the relationship between the key elements. Since the node variables within the Bayesian network are all causal, the main task of this link is to determine the causality of the key elements and draw them with directed edges to form a complete emergency scenario network. Although causality is the most common relationship in the real world, due to the complex evolution process of emergency scenarios and the many situational elements involved, the determination of the causal relationship between various elements is also a complex project, which also needs to be completed with the knowledge of field experts and some technical means.

According to the key elements of the situation in the analysis of the situation evolution law, they are divided into four categories: Situational status (S), emergency activities (B), external environment (H), and emergency resources (M). Among these four types of elements, emergency activities and external environment act on the state of the situation as the reasons, that is to say, the state of the situation is the result, and emergency resources act as the constraint of the interaction between them. Therefore, the state of the situation, the causal relationship between emergency activities and the state of the situation are mainly analyzed in this paper (as shown in Fig. 6.11), and emergency resources are not considered in the construction and solution of the scenario network in this paper.

Fig. 6.11

A Bayesian network at some point

The evolution of the emergency scenario is a continuous, dynamic process. Assuming that the whole evolution of the emergency scenario is divided into T points, denoted as t₀, t₁, t₂, … t_n respectively, and T₀ is the initial moment of the occurrence, the dynamic Bayesian network can be represented in Fig. 6.12.

Fig. 6.12

Dynamic Bayesian networks for the evolution of the emergency scenario

In Fig. 6.12, as for the initial scenario state S₀, it enters the state S₁ after the node variables B₁ and H₁ have been entered, then S₀, B₂ and H₂ become the input node variables of S₁, and so on. It can be seen that the whole dynamic Bayesian network can be divided into two parts from the perspective of input variables. One part is the initial scenario network, the situation state is S₀, only with nodes B and H as input variables. The other part is the development scenario network. Starting from the situation state S₁, there are three input node variables, namely emergency activity B, external environment H and the situation state S of the preceding moment.

• (3) Situation analysis of power grid disaster emergencies and scenario deduction

After the construction of the emergency scenario network is completed, the scenario can be deduced. The so-called scenario deduction is to predict the scenario state of the next moment according to the scenario state of the current moment. Due to the uncertainty of the scenario evolution, the situation state at the next moment can only be expressed by probability. If the variables of the current scenario state are denoted as e S₁, e S₂, e S₂, e S₂, … e S_n, the input scenario variables are denoted as i S₀, i S₁, i S₂, … i S_k, and the output scenario variables are denoted as o S₀, o S₁, o S₂, …, o S_i etc., the emergency scenario deduction model based on the dynamic Bayesian network is as follows, as shown in Fig. 6.13.

Fig. 6.13

Emergency scenario deduction model

In Fig. 6.13, as for the scenario state variable e S₁, its input scenario variables are i S₀, i S₁, i S₂, …, i S_k etc. With the prior probability for all input scenario variables, namely P(i S₀), P(i S₁), P(i S₂), …, P(i S_k), etc., and the conditional probability, namely P(e S₁ | i S₀, i S₁, i S₂, …, i S_k), then P(e S₁) can be calculated, and so on. P(e S₁), P(e S₂), …, P(e S_n) can be calculated respectively. If i S₀, i S₁, e S₁, e S₂ act as the input scenario variables, the probability of the next moment scenario variable o S₀, namely P(o S₀), can be deduced based on the conditional probability P(o S₀ | i S₀, i S₁, e S₁, e S₂), and so on. And the probability of P(o S₀), P(o S₁), P(o S₂), P(o S_i), etc. can be calculated respectively, which means a scenario deduction is completed.

In the process of scenario deduction, there are two probabilities to be set in advance. One is the prior probability of the input scenario node variable without a parent node; the other is the scenario node variable with a parent node which needs to estimate the conditional probability. Whether the setting of these two probabilities is reasonable directly determines the accuracy of the final deduction result. In general, the prior probabilities are obtained based on historical experience or historical statistics, while the conditional probabilities need to be given by expert estimates.

• (4) Determination and calculation method of scenario deduction probability

After the construction of emergent scenario dynamic Bayesian network is completed, in order to successfully realize the state of deduction, we must first determine the prior probability of part of the network node variable or expert estimate probability, and then use the prior probability or expert estimate probability to calculate the state of probability, so as to deduce the probability of the next scenario state, complete the process of the scenario deduction. Specifically, it can be divided into two steps: the determination of prior probability or expert estimation probability and the calculation of scenario state probability.

• 1) Determination of the prior probability of some network node variables

The network node variables that need to determine the prior probability or the expert estimate probability include two aspects: first, the prior probability for the node variable without the parent node is determined, and the conditional probability needs to be determined for the node variable with the parent node.

• a) Determination of the prior probability

The so-called prior probability refers to the probability obtained based on historical experience or analysis, which belongs to the unconditional probability. In Fig. 6.12, the node variables B_i and H_i (i = 1, 2, … n) have no parent nodes, so their prior probability needs to be determined in advance, i.e., the probability distribution of P(B_i) and P(H_i) (i = 1, 2, …, n) needs to be determined in advance.

• b) Determination of the expert estimation probability

As for those node variables with a parent node, such as the node variable Si in Fig. 6.12 (i = 0, 2, … n), you need to determine the conditional probability based on historical empirical data or expert estimation methods. Due to the particularity of emergencies, there may be less historical experience data for reference, so more expert estimation methods are used. In order to overcome the limitations of expert knowledge and the influence of personal preferences, the arithmetic mean of the estimated results of multiple experts is generally used as the final estimation result.

• 2) The calculation method of scenario state probability

In order to illustrate the calculation method of scenario state probability, we take Fig. 6.12 as an example to intercept the network node variables at the points of t₀ and t₁ to illustrate the calculation method of scenario state variable S₀ and S₁.

• a) Calculation of the initial scenario state probability P(S₀) (time t₀)

Step 1: Determine the types of network node variables and their values set, as shown in Table 6.1

Table 6.1 Collection of variable types and values of network nodes

Step 2: Determine P(B₁) and P(S₀ | B₁, H₁).

hypothesis:

$$\begin{array}{cc}& \mathrm{P}({B}_{1}=\mathrm{T})=0.9,\mathrm{P}({B}_{1}=\mathrm{F})=0.1;\\ & \mathrm{ P}({\mathrm{H}}_{1}=\mathrm{Pos})=0.7,\mathrm{P}({\mathrm{H}}_{1}=\mathrm{Neg})=0.3;\\ & \mathrm{ P}({\mathrm{S}}_{0}=\mathrm{T}\mid {\mathrm{B}}_{1}=\mathrm{T},\mathrm{H}=\mathrm{Pos})=0.8;\\ & \mathrm{ P}({\mathrm{S}}_{0}=\mathrm{T}\mid {\mathrm{B}}_{1}=\mathrm{T},{\mathrm{H}}_{1}=\mathrm{Neg})=0.7;\\ & \mathrm{ P}({\mathrm{S}}_{0}=\mathrm{T}\mid {\mathrm{B}}_{1}=\mathrm{F},{\mathrm{H}}_{1}=\mathrm{Pos})=0.5;\\ & \mathrm{ P}({\mathrm{S}}_{0}=\mathrm{T}\mid {\mathrm{B}}_{1}=\mathrm{F},{\mathrm{H}}_{1}=\mathrm{Neg})=0.3;\end{array}$$

Step 3: Calculate P(S₀).

According to the full probability formula, there are:

$$\mathsf{P}({\mathsf{S}}_{0}=\mathsf{T})=\mathsf{P}({\mathsf{B}}_{1},{\mathsf{H}}_{1})*\mathsf{P}({\mathsf{S}}_{0}=\mathsf{T}\mid {\mathsf{B}}_{1},{\mathsf{H}}_{1})$$

(6.8)

Since the premise of the Bayesian network assumes that conditional independence,

$$\mathsf{P}(1,{\mathsf{H}}_{1})=\mathsf{P}({\mathsf{B}}_{1})*\mathsf{P}({\mathsf{H}}_{1})$$

(6.9)

Therefore, bring formula (6.9) into formula (6.8)

$$\mathsf{P}({\mathsf{S}}_{0}=\mathsf{T})=\mathsf{P}({\mathsf{B}}_{1})*\mathsf{P}({\mathsf{H}}_{1})*\mathsf{P}({\mathsf{S}}_{0}=\mathsf{T}\mid {\mathsf{B}}_{1},{\mathsf{H}}_{1})$$

(6.10)

Further expansion gives you that:

$$ \begin{aligned} P(S_{0} = T)\, = & \,({\text{B}}_{1} = T)*P({\text{H}}_{1} = {\text{Pos}})*P({\text{S}}_{0} = T|{\text{B}}_{1} = T,{\text{H}}_{1} = {\text{Pos}}) \\ & \quad + P({\text{B}}_{1} = T)*P({\text{H}}_{1} = {\text{Neg}})*P({\text{S}}_{0} = T|{\text{B}}_{1} = T,{\text{H}}_{1} = {\text{Neg}}) \\ & \quad + \mathsf{P}(\mathsf{B}_{1} = \mathsf{F})*\mathsf{P}(\mathsf{H}_{1} = \mathsf{Pos})*\mathsf{P}(\mathsf{S}_{0} = \mathsf{T}|\mathsf{B}_{1} = \mathsf{F},\mathsf{H}_{1} = \mathsf{Pos}) \\ & \quad + P\left( {{\text{B}}_{1} = {\text{F}}} \right)*P\left( {{\text{H}}_{1} = {\text{Neg}}} \right)*P\left( {{\text{S}}_{0} = {\text{T}}|{\text{B}}_{1} = {\text{F}},{\text{H}}_{1} = {\text{Neg}}} \right) \\ & = 0.9*0.7*0.8 + 0.9*0.3*0.7 + 0.1*0.7*0.5 + 0.1*0.3*0.3 \\ & = 0.504 + 0.189 + 0.035 + 0.009 \\ & = 0.737 \\ \end{aligned} $$

According to the calculation of: P(S₀ = F) = 0.263.

• b) t₁ Calculation of situational state probability at time P(S₁)

Step 1: Determine the type of network node variables and their set of values, as above.

Step 2: Determine it P(B₂), P(H₂) as well as P(S₁ | S₀, B₁, H₁).

According to the above calculation results are:

$$P({S}_{0}=\mathrm{T})=0.737,P({S}_{0}=\mathrm{F})=0.263$$

hypothesis: $\mathrm{P}\left({\mathrm{B}}_{2}=\mathrm{T}\right)=0.92$, $\mathrm{P}\left({\mathrm{B}}_{2}=\mathrm{F}\right)=0.08$;

$$\mathrm{P}\left({\mathrm{H}}_{2}=\mathrm{Pos}\right)=0.75,\mathrm{ P}\left({\mathrm{H}}_{2}=\mathrm{Neg}\right)=0.25;$$

$$\mathrm{P}\left({\mathrm{S}}_{1}=\mathrm{T}|{\mathrm{B}}_{2}=\mathrm{T},{\mathrm{H}}_{2}=\mathrm{Pos},{\mathrm{S}}_{0}=\mathrm{T}\right)=0.8;$$

$$\mathrm{P}\left({\mathrm{S}}_{1}=\mathrm{T}|{\mathrm{B}}_{2}=\mathrm{T},{\mathrm{H}}_{2}=\mathrm{Pos},{\mathrm{S}}_{0}=\mathrm{F}\right)=0.65;$$

$$\mathrm{P}\left({\mathrm{S}}_{1}=\mathrm{T}|{\mathrm{B}}_{2}=\mathrm{T},{\mathrm{H}}_{2}=\mathrm{Neg},{\mathrm{S}}_{0}=\mathrm{T}\right)=0.7;$$

$$\mathrm{P}\left({\mathrm{S}}_{1}=\mathrm{T}|{\mathrm{B}}_{2}=\mathrm{T},{\mathrm{H}}_{2}=\mathrm{Neg},{\mathrm{S}}_{0}=\mathrm{F}\right)=0.45;$$

$$\mathrm{P}\left({\mathrm{S}}_{1}=\mathrm{T}|{\mathrm{B}}_{2}=\mathrm{F},{\mathrm{H}}_{2}=\mathrm{Pos},{\mathrm{S}}_{0}=\mathrm{T}\right)=0.5;$$

$$\mathrm{P}\left({\mathrm{S}}_{1}=\mathrm{T}|{\mathrm{B}}_{2}=\mathrm{F},{\mathrm{H}}_{2}=\mathrm{Pos},{\mathrm{S}}_{0}=\mathrm{F}\right)=0.4;$$

$$\mathrm{P}\left({\mathrm{S}}_{1}=\mathrm{T}|{\mathrm{B}}_{2}=\mathrm{F},{\mathrm{H}}_{2}=\mathrm{Neg},{\mathrm{S}}_{0}=\mathrm{T}\right)=0.35;$$

$$\mathrm{P}\left({\mathrm{S}}_{1}=\mathrm{T}|{\mathrm{B}}_{2}=\mathrm{F},{\mathrm{H}}_{2}=\mathrm{Neg},{\mathrm{S}}_{0}=\mathrm{F}\right)=0.2;$$

Step 3: Calculate P(S₁).

According to the full probability formula, there are:

$$\mathsf{P}({\mathsf{S}}_{1}=\mathsf{T})=\mathsf{P}({\mathsf{B}}_{2},{\mathsf{H}}_{2},{\mathsf{S}}_{0})*\mathsf{P}({\mathsf{S}}_{1}=\mathsf{T}\mid {\mathsf{B}}_{2},{\mathsf{H}}_{2},{\mathsf{S}}_{0})$$

(6.11)

Similarly, since the premise of the Bayesian network assumes conditional independence.

$$P({B}_{2},{\mathrm{H}}_{2},{\mathrm{S}}_{0})=P({B}_{2})*P({H}_{2})*P({S}_{0})$$

(6.12)

Therefore, bring formula (6.12) into formula (6.11):

$$\mathsf{P}({S}_{1}=\mathsf{T})=\mathsf{P}({B}_{2})*\mathsf{P}({H}_{2})*\mathsf{P}({S}_{0})*\mathsf{P}({S}_{1}=\mathsf{T}\mid {\mathsf{B}}_{2},{\mathsf{H}}_{2},{\mathsf{S}}_{0})$$

(6.13)

Equation (6.13) further follows:

$$ \begin{gathered} {\text{P}}\left( {{\text{S}}_{1} = {\text{T}}} \right)\, = {\text{P}}\left( {{\text{B}}_{2} = {\text{T}}} \right)*{\text{P}}\left( {{\text{H}}_{2} = {\text{Pos}}} \right)*{\text{P}}\left( {{\text{S}}_{0} = {\text{T}}} \right)*{\text{P}}\left( {{\text{S}}_{1} = {\text{T}}|{\text{B}}_{1} = {\text{T}},{\text{H}}_{1} = {\text{Pos}},{\text{S}}_{0} = {\text{T}}} \right) \hfill \\ + \,{\text{P}}\left( {{\text{B}}_{2} = {\text{T}}} \right)*{\text{P}}\left( {{\text{H}}_{2} = {\text{Pos}}} \right)*{\text{P}}\left( {{\text{S}}_{0} = {\text{F}}} \right)*{\text{P}}\left( {{\text{S}}_{1} = {\text{T}}|{\text{B}}_{2} = {\text{T}},{\text{H}}_{2} = {\text{Pos}},{\text{S}}_{0} = {\text{F}}} \right) \hfill \\ + \,{\text{P}}\left( {{\text{B}}_{2} = {\text{T}}} \right)*{\text{P}}\left( {{\text{H}}_{2} = {\text{Neg}}} \right)*{\text{P}}\left( {{\text{S}}_{0} = {\text{T}}} \right)*{\text{P}}\left( {{\text{S}}_{1} = {\text{T}}|{\text{B}}_{2} = {\text{T}},{\text{H}}_{2} = {\text{Neg}},{\text{S}}_{0} = {\text{T}}} \right) \hfill \\ + \,{\text{P}}\left( {{\text{B}}_{2} = {\text{T}}} \right)*{\text{P}}\left( {{\text{H}}_{2} = {\text{Neg}}} \right)*{\text{P}}\left( {{\text{S}}_{0} = {\text{F}}} \right)*{\text{P}}\left( {{\text{S}}_{1} = {\text{T}}|{\text{B}}_{2} = {\text{T}},{\text{H}}_{2} = {\text{Neg}},{\text{S}}_{0} = {\text{F}}} \right) \hfill \\ + \,{\text{P}}\left( {{\text{B}}_{2} = {\text{F}}} \right)*{\text{P}}\left( {{\text{H}}_{2} = {\text{Pos}}} \right)*{\text{P}}\left( {{\text{S}}_{0} = {\text{T}}} \right)*{\text{P}}\left( {{\text{S}}_{1} = {\text{T}}|{\text{B}}_{2} = {\text{F}},{\text{H}}_{2} = {\text{Pos}},{\text{S}}_{0} = {\text{T}}} \right) \hfill \\ + \,{\text{P}}\left( {{\text{B}}_{2} = {\text{F}}} \right)*{\text{P}}\left( {{\text{H}}_{2} = {\text{Pos}}} \right)*{\text{P}}\left( {{\text{S}}_{0} = {\text{F}}} \right)*{\text{P}}\left( {{\text{S}}_{1} = {\text{T}}|{\text{B}}_{2} = {\text{F}},{\text{H}}_{2} = {\text{Pos}},{\text{S}}_{0} = {\text{F}}} \right) \hfill \\ + \,{\text{P}}\left( {{\text{B}}_{2} = {\text{F}}} \right)*{\text{P}}\left( {{\text{H}}_{2} = {\text{Neg}}} \right)*{\text{P}}\left( {{\text{S}}_{0} = {\text{T}}} \right)*{\text{P}}\left( {{\text{S}}_{1} = {\text{T}}|{\text{B}}_{2} = {\text{F}},{\text{H}}_{2} = {\text{Neg}},{\text{S}}_{0} = {\text{T}}} \right) \hfill \\ + \,{\text{P}}\left( {{\text{B}}_{2} = {\text{F}}} \right)*{\text{P}}\left( {{\text{H}}_{2} = {\text{Neg}}} \right)*{\text{P}}\left( {{\text{S}}_{0} = {\text{F}}} \right)*{\text{P}}\left( {{\text{S}}_{1} = {\text{T}}|{\text{B}}_{2} = {\text{F}},{\text{H}}_{2} = {\text{Neg}},{\text{S}}_{0} = {\text{F}}} \right) \hfill \\ = 0.{\text{92}}*0.{\text{75}}*0.{\text{737}}*0.{\text{8}} + 0.{\text{92}}*0.{\text{75}}*0.{\text{263}}*0.{\text{65}} \hfill \\ + \,0.{\text{92}}*0.{\text{25}}*0.{\text{737}}*0.{\text{7}} + 0.{\text{92}}*0.{\text{25}}*0.{\text{263}}*0.{\text{45}} \hfill \\ + \,0.0{\text{8}}*0.{\text{75}}*0.{\text{737}}*0.{\text{5}} + 0.0{\text{8}}*0.{\text{75}}*0.{\text{263}}*0.{\text{4}} \hfill \\ + \,0.0{\text{8}}*0.{\text{25}}*0.{\text{737}}*0.{\text{35}} + 0.0{\text{8}}*0.{\text{25}}*0.{\text{263}}*0.{\text{2}} \hfill \\ = 0.{\text{4}}0{\text{6824}} + 0.0{\text{9}}0{\text{735}} + 0.{\text{1}}0{\text{9164}} + 0.0{\text{27221}} + \hfill \\ + \,0.0{\text{2211}} + 0.00{\text{6312}} + 0.00{\text{5159}} + 0.00{\text{1}}0{\text{52}} \hfill \\ = 0.{\text{669}} \hfill \\ \end{gathered} $$

According to the calculation of: $\mathrm{P}({S}_{1}=\mathrm{F})=0.331.$

6.2 3D Visualization Command Technology

6.2.1 3D Visualization Technology

Compared with the traditional 3D visualization system, the major problems of the large-scale 3D visualization system lie in the huge amount of data, the transmission waiting time beyond the acceptable range, being very suitable for external storage drawing; the large-scale 3D visualization system is mainly used in large projects such as smart city and field security. Its requirements for reliability is high. It requires convenient and stable data docking services, and the system design and implementation are relatively complicated. The main performance evaluation indexes of a large-scale 3D visualization system include initial model waiting time, initial transmission data amount, system CPU occupancy rate, system memory footprint, page completion time and so on.

• (1) Data segmentation technology

The idea of data segmentation is to decompose a large-scale 3D scene model into a collection of slices with a quadtree or octree structure. The root node of the tree structure is a version of the scene with less detail. The child node below the root node divides the whole scene into four or eight blocks of the same size, for a representation with more details. Each of these blocks will also have four or eight subblocks, thus further dividing the entire scene. Large-scale terrain and scenes can be rendered with segmented data. Theoretically, the total amount of data is no longer limited. At the same time, the screen spatial error of the scene can be accurately controlled and the GPU can be better utilized. In the tree structure, the geometric error is twice that of the next layer with less high precision.

Data segmentation has achieved great success in virtual earths, games and other applications that require large-scale scene rendering. Even though the GPU is improved and the hardware capabilities are improved, with the development of scene scale, the data segmentation idea is still very suitable for large-scale rendering. The process of splitting the data is not related to its rendering process, and in practice, slices can be created from any scene model. But the slice must have the following characteristics: Uniform segmentation: Balance is crucial for the final generated scene tree structure. Otherwise, it will lose the advantage of tree structure and decrease the performance sharply.

Geometric error with monotonicity: For each slice, the greatest geometric error must be known. The geometric error is the maximum of the difference between all nodes in the full detail model and the nearest corresponding point in the detail reduction model, which must depend monotonically on the associated slice hierarchy. That is, rendering the sub-slices of all A slices will get A smaller error compared to rendering only the slice. Geometric errors were calculated as the slice was created.

Known enclosure box: Each slice must have a known enclosure box. The enclosure box is used together with the geometric error to calculate the screen space error of all pieces of A.

A quadtree is a tree-like structure, with four subblocks at each node. It divides the 2D spatial data into four equal subspaces, and so on, until the recursive termination condition is reached. The structure of the quadtree is relatively simple. The structure of the conventional quadtree is shown in Fig. 6.14. The leaf node stores spatial data, the intermediate node and the root node do not store data.

Fig. 6.14

Schematic representation of the quadruple tree

The octree is a generalisation of the quadtree in 3Ds, a tree-like structure used to manage and describe 3Ds. Each node of the octree has eight child nodes, and the combination of the spatial division of child nodes is the spatial division of the parent nodes. The octree structure can be expressed as the following steps: (1) Set the maximum recursive depth or minimum segmentation size. (2) Find the x, y, z direction boundary of the scene to establish the first cube, that is, the root node. (3) Put 3D spatial metadata into a cube that can be included without child nodes. (4) If the recursive termination condition is not reached, continue eight equal points and allocate all the spatial data contained in the node to the eight child nodes. (5) If it is found that the data assigned by the child node is not zero and is the same as the data of the parent node, the child node stops subdivision to avoid infinite segmentation. (6) Repeat process 3 until the maximum recursive depth is reached. The whole process is shown in Fig. 6.15.

Fig. 6.15

Schematic diagram of the octree segmentation process

• (2) Dynamic scheduling technology

The 3D scene model of a large-scale 3D visualization system has a huge amount of data. Although compression can greatly reduce the storage space, it is far from enough to store the local memory. No matter how the hardware conditions develop, this is not a good solution. Moreover, the space span is large, loading too much useless data will undoubtedly cause unnecessary waste of resources, and will lead to long startup time of the application. In external memory drawing technology, only a small subset of the data set is in the system memory at any time, and the rest of the data is temporarily stored in external memory, such as a hard disk or network server. According to the perspective parameter, the new datasets are loaded into the system memory, and the expired datasets are removed from the memory, which is the process of dynamic scheduling. The goal of dynamic scheduling is to seamlessly load the required new data into the working data set by using the external memory drawing technology.

In order to dynamically schedule the node slices in the tree structure, it is first necessary to set the maximum screen spatial error rho that can be tolerated, and the rho value can be adjusted to achieve good display results at different hardware levels. Rendering from the root node, the screen space error at this time is calculated by the maximum geometric error of the node, compared with rho. This is a recursive process until the lowest requirement for screen space error is met. Generally, to take advantage of GPU depth buffering by accessing child nodes from the front to the back, this is very efficient and easy to implement.

• (3) Display method design

Usually, the goal is to render a better scene with the simplest LOD, so it theoretically requires a standard to determine whether a LOD rendering result will make the scene look better. A useful objective quality measure is the difference in pixel units, also called the number of pixels in screen spatial error (Screen Space Error). It is usually difficult to calculate this value accurately, but it can be estimated by simplifying the estimation. In the figure, consider the LOD for simplifying the object, let the observer distance from the object along the viewpoint direction be d, the optic vertebral width be w, the display resolution be x pixels, and the field angle be θ. The geometric error of the simplified version of the object is ε, meaning that each vertex on the full detail model deviates from the nearest corresponding vertex on the low detail model in no more than ε units (the geometric error is calculated when the slice is created. The finest layer geometric error is selected when the slice tree is created, the selected error is 0.5. The next geometric error is 2 times the previous one). The screen space error is ρ, and a simplified formula of ρ can be derived, which shows how much ρ is, the next detailed version of the object will be rendered.

Fig. 6.16

Schematic representation of the spatial error

From Fig. 6.16, we can see that ρ is proportional to ε, it follows that:

$$\begin{array}{cc}& \frac{\epsilon }{\omega }=\frac{\rho }{x}\\ & \\ & \rho =\frac{\epsilon x}{\omega }\end{array}$$

(6.14)

The optic vertebral width W is easily obtained at distance d, substituting with the upper formula:

$$ \begin{array}{*{20}c} {\omega = 2d{\text{tan}}\frac{\theta }{2}} \\ {\rho = \frac{\varepsilon x}{{2d{\text{tan}}\frac{\theta }{2}}}} \\ \end{array} $$

(6.15)

As can be seen from the above equation, the screen spatial error p is estimated by the distance between the observer and the object, the viewpoint parameters and the geometric error ε of the object ε. When the perspective is certain and the distance is certain, the larger the geometric error of the object, the greater the screen spatial error introduced. Therefore, we can judge whether the current LOD is fine enough according to the screen spatial error introduced by the object in the runtime. Otherwise, it can be further refined. Usually during the rendering process, the enclosure box of the object is used together with the geometric error to calculate the screen space error of all pieces of A. The general rendering engine will set the maximum screen space error that can be tolerated, and dynamically adjust according to the state of the hardware resource. If the maximum allowable range is exceeded, the child node slices are loaded for further refinement.

The rationale for dynamically selecting slices for rendering with HLOD ideas is described above, and an algorithm will be used to describe this dynamic selection process below. First, set the maximum screen space error that the rendering system can tolerate to rho, and the RHO value can vary. At the same time, enter the instance that needs to have slices and the current scene state, including the distance from the slice to the camera, screen size, resolution, viewing angle range, etc. For each slice, the introduced screen space error was calculated based on the maximum geometric error of the slice and then compared with the rho values. If the screen space error of the slice is less than the allowed upper bound, the slice is rendered. Otherwise, its eight child nodes recursively call the function. The process continues until the entire scene can meet the lowest layer of detail of the screen space error requirements.

In a rendering function, subslices are requested for rendering only if the viewpoint parameters change and the scene needs more details. Due to the delay of slice network transmission and rendering, the 3D scene jitter will be caused, and the picture is not smooth enough when the viewpoint is moved. Using the caching mechanism, the prefetch optimization strategy is developed.

Prefetch optimization—After initializing the loading, continue to request the sliced child nodes in the current viewport, and preload the data to the browser cache. Using the browser negotiation cache mechanism, when the viewport is moved, the new slice can be loaded directly from the browser cache, reducing the download time and increasing the efficiency.

Rendering loads the first section that meets the screen space error requirements first, but in the optimized algorithm, the subsection of each rendered section is also requested to be loaded, which is a pre-fetch slice procedure, because the likelihood that the corresponding subsection needs to load increases when the observer moves. In addition, when the subslice of the current slice is not loaded, regardless of the small screen error, the current slice will be stained. Therefore, when a block of cache is lost, the system renders the optimal data currently available.

6.2.2 Visualization Command Technology

• (1) Map visualization technology

At present, map visualization has relatively mature technologies, such as state Grid unified GIS platform within the system, WebGIS technology outside the system, Amap, Baidu Map, etc.

• 1) Unified GIS platform of State Grid

Due to the natural connection between power grid equipment and facilities and geospatial location, GIS has been widely used in the power industry in recent years as an important tool, technology and means to obtain, store, analyze, and manage geospatial data.

The State Grid Unified GIS Platform is a GIS application platform with independent intellectual property rights based on a home-made GIS platform. The design of the platform has made a lot of innovations in architecture and key technologies to meet the characteristics of power applications. The platform is designed with a flexible architecture that can run without relying on specific base GIS platform software; it makes full use of multi-level caching technology to reduce the network and server pressure caused by centralised spatial data deployment, and achieve the high performance service for the concurrent maintenance of network graphics and topology by a large number of users.

In terms of data scale, the power grid GIS platform integrates and maintains massive basic geographic data, power grid spatial data, and power grid topology data. On the business application scale, the platform has gradually realized the integration with production management, marketing management, distribution automation, power transmission and transformation status monitoring, grid planning system, providing grid resources graphics visualization and space analysis services for the grid production management, disaster prevention and mitigation and emergency command, planning and design, power Internet of things, power communication and other kinds of business applications.

The State Grid Unified GIS Platform API provides a very complete map function, such as distance viewing, search prompts, field of view retrieval, map zoom and a series of practical functions, which are closely related to the visual display of maps in this article.

• 2) WebGIS technology

WebGIS (Network Geographic Information System) is the release and application of geospatial data through the Internet to realize the sharing and interoperation of spatial data, such as online query and business processing of GIS information. The WebGIS client-side uses a Web browser, such as IE, FireFox. WebGIS is a new technology that uses Internet technology to expand and improve GIS. Its core is the application system of HTTP standard embedded in GIS, so as to realize the management and release of spatial information under Internet environment. WEBGIS can use multi-host, multi-database for distributed deployment, and can be connected through Internet/Intranet. It is a browser/server (B/S) structure. The server-side provides information and services to the client-side, and the browser (client-side) is able to obtain various spatial information and applications.

The information in WebGIS is mainly the spatial data that needs to be expressed in terms of graphics and images. Users can query and analyze the spatial data through interactive operation. Users can easily browse various meteorological data based on WebGIS undermap, and carry out various meteorological information retrieval and spatial analysis. Using WebGIS technology, basic observation, basic products, service products and basic geographic information are displayed in layers, graphics, images and other ways, so as to provide users with a variety of more intuitive data sharing services, including time, space, meteorological data types, observation tools, layers, etc.

• 3) Leaflet technology

Leaflet is a modern, open source JavaScript library developed for mobile devices. The design of Leaflet adheres to the philosophy of simplicity, high performance and good usability. It operates efficiently on all major desktop and mobile platforms. It can take advantage of HTML5 and CSS 3 in modern browsers, and also supports old browser access. The powerful open source library plug-ins of Leaflet involve all aspects of map application, including map services, data supply, data format, geographic coding, route and route search, map controls and interaction, etc. More than 140 plug-ins of many different types are involved. At the same time, it supports custom controls, with good scalability.

Leaflet is a lightweight and cross-platform, with a compressed library package of 38kB. It is mobile friendly and all effects on the PC are seamlessly rendered on mobile terminals. Leaflet provides an L.Icon interface to set the icon image and shadow image to make the icon more three-dimensional and to set the position of the anchor point to make the icon positioning more accurate; it also supports the implementation of temporal animation, heat map and aggregation point effects.

Leaflet uses html5 to achieve high performance rendering. The map rendering is fine and smooth, and the combination with SuperMap iClient9D for Leaflet can easily render more than 100,000 points of data, while supporting vector chunking layers. The vector chunking is a client-side vector map, and the rendering of the map is all done on the client side. The server provides the map vector data, as well as information on the rendering style attributes of the map. SuperMap also provides various services such as network analysis, spatial analysis, traffic interchange, real-time data services, big data analysis and so on.

GeoJSON is a structural format for web to transfer and encode a variety of geographic data. Leaflet provides the L.geoJSON interface to easily encode GeoJSON and render it on the map; OGC services are also very comprehensive. Based on the L.TileLayer interface, it can be docked on WMS, WMTS and other services. At the same time, L.TileLayer can be extended to build custom layers.

• (2) SVG visualization technology

• 1) Overview of SVG technology

SVG is a language based on XML to describe 2D graphics. It uses the descriptive language of text format to describe the content of images, and it is a vector graphics format, independent of image resolution. SVG is easy to edit and modify, and image files are readable. It can interact with existing technologies, and can control its objects by embedding JavaScript scripts. The SVG graphic format can be easily indexed to search the content of the image. SVG can use three types of graphic objects: images, text, and vector graphic shapes (paths consisting of lines and curves). At the same time, we can transform, marshal, stylize and combine the graphic objects into new objects. SVG can be customized, including nested transformations, paths of clips, template objects and filtering effects. SVG can also be drawn dynamically and interactively. In addition, SVG can define animations, triggered either by script or declaratively (embedding the animated elements of SVG into SVG content). With the addition of the JavaScript language, the SVG text object module (DOM, Document Object Model) provides access to all of its attributes, elements, and features, and can handle very complex and cumbersome functionality. SVG is very rich in event handling and can support various event attributes in HTML, such as mouse events such as onmouseover (mouse movement), onclick (mouse click), and keyboard events such as onkeydown (keyboard down), onkeyup (keyboard up), and can also define these event attributes to any SVG graphics object. Because of its compatibility and leverage of other Web standards, features such as scripting involving HTMLS and SVG elements can be done simultaneously within the same Web page.

• a) SVG fully supports the DOM (document element model), so external events can be controlled by scripting languages such as mouse manipulation, keyboard manipulation, and manipulation of various images and elements.

• b) SVG images are in a text-based format, so all the text in SVG can be searched and queried through a web search engine, or directly edited and searched through a browser.

• c) Command statements in SVG images can interact freely with scripts or programs, such as JavaScript or XML, and can be implemented entirely by code.

• d) Since SVG is not restricted by the operating system, SVG can work cross-platform and solve the problems related to color, bandwidth, external output, etc.

• e) SVG image files can be easily and dynamically generated by a computer programming language, such as JavaScript, Java, etc. This is a great advantage for some graphs that need to be dynamically displayed, because the images can be displayed in real-time and dynamically according to the data in the database.

• 2) Introduction to SVG Graphics Objects

SVG provides a rich set of graphic objects including rectangles < rect >, circles < circle >, ellipses < ellipse >, lines < line >, folds < polyline >, polygons < polygon >, and so on. And you can make these graphics objects display different effects by setting different properties and display styles. SVG also provides group management tags (< g > tags), definition defs references, and style definitions for the same attributes of the same class of objects, so that objects of the same type can be set to the same attributes without the unnecessary hassle of multiple definitions.

Objects in SVG files are represented using XML tags as they are in XML. In the visual system, the control of SUG is through traversing the ID of the object in the SUG file, and after finding the object, we can dynamically change its style, whether it is visible or not, to achieve the effect of SUG graphics dynamic display.

• 3) Research on SVG graphics visualization technology

SVG display method is mainly through the SVG graphics files embedded in the HTML page for display, the method can be achieved mainly through the following ways:

• a) < embed >:

  Syntax: < embed src = “circlel.svg” type = “image/svg + xml"/ >

  Advantage: The < embed > tag supports all major browsers and can be scripted.

  Disadvantages: Not recommended in HTML 4 and XHTML, but allowed in HTML.

• b) < object >:

  Syntax: < object data = “circlel.svg” type = “image/svg + xml” >

  Advantages:The < object > tag supports all major browsers and supports the HTML4, XHTML, and HTMLS standards.

  Disadvantages: Scripts are not allowed.

• c) < iframe >:

  Syntax: < iframe src = “circlel.svg” >

  Advantage: The < iframe > tag supports all major browsers and can be scripted.

  Disadvantages: Not recommended in HTML 4 and XHTML, but allowed in HTML.

The SVG embedding implemented in this project primarily uses the < embed > label. When using the “embed” tag, you can use multiple attributes at the same time to reference an SVG diagram. You can specify the width and height of this SVG diagram in the HTML page by defining the values of the width and height attributes. Type can be used to specify the type of SVG, typically image/svg-xml. Src specifies the specific path address of an SVG data file. Once the “embed” tag is defined, the SVG diagram under that path is displayed in the appropriate position on the HTML page. However, it should be noted that SVG currently does not support GB encoding format, you must use UTF-8 encoding to complete the Chinese font display. A more significant feature of the Src attribute is the ability to dynamically load different SVG graphics without changing the page code by dynamically setting the property values.

The refreshing mechanism of SVG is realized by timing reloading of SVG browsing, so we can directly use the Java Script timer setTimeout or setInterval to get the data in the database in real-time through AJAX data acquisition technology, and change the attribute value of the specified graphics in the SVG image or change the text in a text box through the data acquisition.

• (3) Echarts visualization technology

ECharts, a pure Javascript charting library, can run smoothly on PCs and mobile devices, compatible with most current browsers (IE8/9/10/11, Chrome, Firefox, Safari, etc.), and relies on the lightweight Canvas class library ZRender to provide intuitive, vivid, interactive, highly customizable data visualization charts. ZRender (Zlevel Render) is a lightweight Canvas class library, MVC encapsulated, data-driven, and provides a DOM-like event model.

ECharts provides regular line charts, bar charts, scatter charts, pie charts, k line charts, box charts for statistics, maps, thermal charts, line charts for geographic data visualization, diagrams for data relationship visualization, treemap, parallel coordinates for multidimensional data visualization, funnels for BI, dashboards, and mashups between diagrams.

ECharts is driven by data, and changes in data drive the changes represented by the chart. As a result, the implementation of dynamic data becomes much simpler. ECharts will find the difference between the two sets of data and animate the changes only if it obtains the data and fill in the data.

The volume of the chart library should be as small as possible. The refactoring of the ECharts and ZRender code resulted in a reduction in core volume. The refactoring of ECharts and ZRender code brings the ability to shrink core volumes to provide on-demand packaging because ECharts has many components and continues to grow.

ECharts 3 began to enhance support for multidimensional data. In addition to common multidimensional visualization tools such as parallel coordinates, for traditional scatter graphs, the incoming data can be multidimensional. With the visual mapping component VisualMap, it can provide rich visual coding, and map data of different dimensions onto different visual channels of different color, size, transparency, shading, etc.

6.2.3 Electric Power Emergency Visualization

Power emergency visualization mainly refers to map visualization, power emergency data visualization and emergency command visualization. Map visualization refers to the visualization of power facilities by using the unified GIS platform of the national network, and the ability to timely and accurately obtain the affected power grid equipment or power facilities before or in the event of a disaster. Power emergency data visualization refers to the visualization of various data in the process of power emergency response, mainly including the data of power outage caused by a disaster (suspended substation, line, station area and users experiencing power outage), emergency repair data (input personnel, input vehicles, input emergency repair team and input emergency repair resources). Emergency command visualization refers to the visualization of emergency response process in the process of emergency response, so that the emergency personnel participating in the emergency response process have an overall understanding of the overall response process and assist in the emergency response work.

The Unified GIS Platform of China Network is a GIS application platform with independent intellectual property based on the research and development of domestic GIS platform. Multi-level caching technology is fully used to reduce the network and server pressure caused by the centralized deployment of spatial data. Power grid GIS platform integrates and maintains massive basic geographic data, power grid spatial data and power grid topology data. The API of the Unified GIS Platform for State Network provides a series of complete and practical map functions, such as distance viewing, searching and displaying, searching in the field of vision, and map scaling. In the application of electric power emergency visualization, the GIS Platform for State Network is mainly used to forecast and count the affected power facilities in time in combination with the prediction scope and impact scope of natural disasters, and to conduct early warning response and emergency response, as shown in Fig. 6.17.

Fig. 6.17

Visualization of natural disaster impact power grid forecast

In the field of electric power emergency response, data visualization mainly includes visualization of spatial information (visualization of emergency response resource distribution, etc.), data visualization (visualization of power grid damage and recovery, visualization of power grid damage prediction, visualization of emergency response resource demand, visualization of meteorological disasters, etc.), and visualization of on-site images (field video monitoring, field video recording of emergencies, etc.). The visualization technologies in respect of three aspects complement each other and can effectively help the staff members to have a direct understanding of the different regional states of the power grid, distribution of emergency response resources, field, and other information.

Power grid damage and recovery: In the form of a histogram, visualize the situation of lines (damaged, restored), substations (damaged, restored), distribution station area (cumulative power outage, restored), users (cumulative power outage households, restored households), important users (cumulative outage households, restored households) and other situation information. At the same time, combined with the actual power grid business, it provides the configuration function of the column chart template information items, colors and other contents, as shown in Fig. 6.18.

Fig. 6.18

Visualization of grid damage and recovery

The power grid emergency repair input function is based on Echarts technology, and displays data such as large-scale emergency repair machinery, emergency lighting equipment, emergency repair teams, emergency repair vehicles, power generation vehicles, generators, etc. in the form of pie charts, radar charts, and text, as shown in Fig. 6.19.

Fig. 6.19

Schematic diagram of grid damage and recovery

Emergency command visualization is mainly based on the emergency handling process and related tasks generated by the special emergency plan in the emergency handling process, so that the emergency response staff has a comprehensive and in-depth understanding of the overall emergency handling process, and they can fully understand the emergency handling process and the tasks of each stage and link, real-time task status feedback, and the emergency command personnel in the emergency command center can fully understand the progress of the emergency handling work and the completion of each work to conduct visual emergency command.

According to the special emergency plan, combined with the departmental disposal plan and the on-site disposal plan, the monitoring and early warning stage is divided into several links “risk monitoring, analysis and judgment, early warning suggestions, early warning review, early warning issuance, early warning action, early warning adjustment or lifting”, and then the corresponding tasks (what to do) and related suggestions (how to do) of each department in each link are accurately pushed, so that the tasks can be accurately pushed to the relevant responsible persons, and the relevant status of each task can be tracked in real-time “not viewed, viewed, in progress, completed”. In the tasks of each department in the corresponding link, you can filter and view your own tasks. At the same time, they can be notified of the need to complete relevant tasks through SMS. If all the tasks in a certain link are “viewed” by all personnel, the whole process can be automatically carried out to the next link. One link can be highlighted and flashed when it is its turn to be carried out.

The response-end stage is divided into several links “response-end, recommendation review, release of information”, and then the corresponding tasks (what to do) and related suggestions (how to do) of each department in each link are accurately pushed, so that the tasks can be accurately pushed to the relevant responsible persons, and the relevant tasks can be tracked in real-time status” Not Viewed, Viewed, In Progress, Completed”. In the tasks of each department in the corresponding link, you can filter and view your own tasks. At the same time, they can be notified of the need to complete relevant tasks through SMS. If all the work tasks in a certain link are “viewed”, the whole process can be automatically carried out to the next link. The link can be highlighted and flashed when it is its turn to be carried out. The process continues until the end of the response, as shown in Fig. 6.20.

Fig. 6.20

Visualization diagram of emergency response

6.3 Integrated Social Information Emergency Command and Decision-Making Technology

6.3.1 Social Information Acquisition and Processing

• (1) Access to information from social sources

Information Extraction (IE): Extracts target factual information from unstructured natural language text and then represents it in the form of structured data. Web information extraction is an important part of information extraction, which is the process of extracting subject content information from web pages and presenting it in a structured manner.

Most of the data on the Internet is presented in the form of HTML documents, which is a kind of semi-structured data, the position and layout of its text, images and other contents are arbitrary and unstructured, but the contents of the web pages are organized according to a certain hierarchical structure and meet the DOM standard set by W3C for operation and access. HTML is a markup language document that lacks semantic information. HTML tags are used to present information data in the browser, but they can only express very limited semantic information. Data on the Internet is characterized by huge volume, heterogeneous data sources, lack of semantic information and dynamic variability, which brings many difficulties to web information extraction. Firstly, the hundreds of millions of web sites on the Internet construct a multi-data source environment, as each site is usually independently laid out and designed, resulting in the heterogeneous nature of multiple data sources; Secondly, in addition to the subject content to be extracted, the HTML document also includes noise content such as navigation bar, advertising area and related recommendations, which should be filtered out during web information extraction. Last but not least, it is the dynamic variability of web data. Internet content is updated and changed every day, especially online social media and news data, which are updated frequently and produce a large amount of data anytime and anywhere, and this part of the content is more data-intensive, interactive and more complex in terms of extraction techniques.

In order to solve the above-mentioned problems in web information extraction, many web information extraction methods have been proposed by researchers at home and abroad, and good results have been achieved in general. According to the differences in the principles of web information extraction and extraction objectives, there are four main categories:

• 1) Rule-based template information extraction method

By observation, although different websites have different design styles and content layouts, for the same web site or a site board, the pages under it have the same or similar structure, which is due to the widespread use of template-population technology in web development, i.e., the front-end pages are often generated by selecting data from the back-end database to populate the same template based on requests. Naturally, if the design template of the web site is somehow obtained and the corresponding information extraction wrapper is generated, then the subject information of the data source can be accurately obtained. As mentioned earlier, because different data sources have different layout structures, a wrapper is only valid for a class of web pages with similar layout structures, so the disadvantage of the method is the high cost of rule maintenance and poor scalability. If you want to handle web pages from different data sources at the same time, you need to design different wrappers for different web sites or web boards, add them to the rule template library and maintain them regularly. The method is implemented with manually written rule templates and (semi-)automated wrapper generation techniques, where automated template generation techniques are used to generate extraction rules from some specific training sets using machine learning techniques. Representative IE systems that use this method are Minerva and WIEN.

• 2) Information extraction technology based on visual features

HTML document is a collection of web tags and web content, where the tags have two main roles: on the one hand, they are used to organize web content, and on the other hand, they provide page display functions, such as page text style, segmentation control, body text and noise distribution area, etc. So the web page has obvious visual features in the content location, style. These visual features provide important clues for the implementation of page segmentation and information extraction, of which the VIPS algorithm proposed by Microsoft Asia Research Institute is the most classic web information extraction algorithm based on visual features. In the early days, the visual block-based web information extraction technology could segment web pages more accurately based on font size, background and other information, but with the development of web technology, web pages have become more diverse and richer, making it more difficult to extract visual features based on heuristic algorithms, resulting in insufficient accuracy of web information extraction.

• 3) Statistical-based information extraction techniques

Because web pages are visually divided into different areas, and in fact, the amount of statistical information such as text content and web page tags in different areas is also unevenly distributed, according to the distribution characteristics of these statistical information, the statistical web information extraction technology designs different strategies to achieve the purpose of web information extraction. The number of Chinese characters in the <table> tag in the body of the web page is used as a measure of the web body region, and the ratio of linked text to normal text in each region of the web page is used to set the body extraction threshold to consider the ratio of the number of characters to the number of hyperlinks in different positions of the web page to distinguish the web body region. The advantage of this approach is that it is not limited by data sources and does not require supervised learning methods to train web pages, which is somewhat generalizable.

• 4) Web information extraction technology based on DOM tree structure

Before extracting information from a web page, a web document can be transformed into a parse tree, also known as a DOM tree, using a web parser. DOM tree can visualize the hierarchical structure of HTML tags, and the hierarchy is closely related to the content of web pages. The web extraction technology based on DOM tree structure is an important technical direction in web information extraction method. Many classical algorithms and mature systems have been developed, such as the MDR algorithm proposed by Liu et al. in the early twenty-first century and the improved algorithm, which mainly extracts data records of <table> table nodes, and the web information extraction method based on tag path clustering was designed by Miao et al. in 2009.

At present, in the design of many web information extraction methods, it is often not limited to a certain extraction technique, such as the web information extraction method based on tag path ratio proposed by Wu Gongqing et al. The algorithm integrates the structure of web pages and statistical information to complete the extraction of web news body.

In the field of information retrieval and statistical classification, accuracy P and recall R are often used to evaluate the quality of results. Similarly, these two metrics can be applied to web information extraction systems to measure the goodness of web information extraction algorithms. Here, the following formula is defined:

$$ P = \frac{IE\,Number\,of\,results\,correctly\,extracted\,by\,the\,system}{{IE\,Number\,of\,all\,extraction\,results\,in\,the\,system}} \times 100\% $$

(6.16)

$$ P = \frac{IE\,Number\,of\,results\,correctly\,extracted\,by\,the\,system}{{IE\,The\,system\,should\,extract\,the\,correct\,number\,of\,results}} \times 100\% $$

(6.17)

According to the definition of the formula, it is obvious that P and R take values from 0 to 1. The higher the value, the higher the accuracy or recall, indicating that the information extraction system performs better. However, for an IE system, these two metrics cannot be obtained at the same time, and the improvement of one metric tends to reduce the other system metric. In order to judge the merits of IE systems more objectively, the F-index has been introduced, which is generally characterized by taking the geometric mean of the accuracy P and the recall R.

• (2) Information resource de-duplication

The web pages of the Internet are linked together by hyperlinks, using the web pages as nodes and the hyperlinks as links to form a huge network graph of document resources. The principle of a web crawler is to request the corresponding HTML document on a site through some initial URL, then parse the content of the HTML document, extract the URL link, and start the next round of web crawling. In this way, we achieve a constant stream of information data crawled from the Internet.

The general process of a traditional web crawler roughly goes through several core parts, including initial configuration, web page download and web page parsing, and its flow chart is shown in Fig. 6.21.

Fig. 6.21

Workflow diagram

Read the web crawler related configuration parameters from the configuration file, such as crawl depth, number of crawler threads, number of failed page retries and URL filtering rules, and then add the manually collected seed URLs to initialize the web crawler URL queue.

The downloading of web pages is done through the downloader in the web crawler, the web downloader usually uses the HTTP protocol and can design the request parameters, the web downloader implementation can be based on the Java’s HTTP client tool.

As for the web pages downloaded to the local area, the web crawler will call the web parser to extract the web page information, which mainly includes two parts. The first is to extract the body information from each web document and store it in a local file or database, and the second is to lift the URL links in the web document and put them into the front URL queue after URL de-duplication.

The web downloader again takes a new URL link from the URL queue and starts the next round of web crawlers until some manual end condition is reached.

In this way, after certain initial conditions are manually set, the web crawler program can independently and continuously complete the collection of web data until the crawler program meets the stopping conditions set by the system.

The traditional web process described above is a batch-type crawler, where the web data collection task is opened with a limited crawl target and crawl range, and stops crawling when the stopping condition is reached. In addition, there are incremental crawlers and vertical crawlers by crawler function. Incremental crawlers are widely used in general search engines to accomplish the task of continuous web collection and regular updates, while vertical crawlers focus only on web information of a specific topic or industry. For the task of collecting web data to be carried out for a long period of time, you should choose the incremental crawler method to continuously collect new web information data and update the old information data.

Due to the link relationship between Internet web pages, the URL extracted from the current web page link may have been downloaded and resolved in the previous web crawler, so it is not desired that the same web page be downloaded multiple times in the subsequent web crawling process, because repeated downloads will not only waste system resources, but also activate the corresponding anti-crawler rules of the site due to the excessive access load to the data collection site, resulting in the subsequent inability to collect web data from the site. So web page de-duplication is an important element in the web data collection process, and there are many related techniques to solve this repetitive downloading problem, such as URL de-duplication techniques from the link perspective, Sim Hash and other algorithms from the web page content. The URL de-duplication technique starts from the hyperlinks of the web pages and selects the links that have not yet been downloaded to the queue of URLs to be crawled, which basically ensures that the same web page will not be repeatedly crawled; The Sim Hash algorithm generates the information fingerprint of a web page through the Hash function, and determines the similarity of the web page by comparing the similarity of the two information fingerprints, thus realizing the purpose of web page de-duplication. Here, we only introduce and analyze the URL de-duplication technique.

It is easy to think that in the crawler system, by creating a global URL library to store the links that have been collected in the historical crawler, and then comparing the URLs of the web pages to be downloaded, we can determine whether this URL corresponds to a web page that has been crawled. The simplest way is to use disk files to store each historical URL link sequentially, and then perform a search to match the new URL link to achieve the purpose of URL de-duplication, but this will become a performance bottleneck for large-scale web crawlers, both in terms of storage space and search efficiency. Considering the efficiency of web crawlers and memory storage space, URL de-duplication technology nowadays is mainly about how to store the URL library and find the URL, and currently the main idea of Hash mapping is used.

• 1) Storage based on MD5 compression mapping

MD5 is an encryption algorithm that uses the Hash Hash idea. It is applied to URL de-duplication by first compressing the link string to get a 128-bit integer information fingerprint, then Hash mapping the information fingerprint to get the physical address of the storage, and by comparing the two URL information fingerprints to see if they are the same, we can determine the duplication of the URL. In addition, it is good that the chance of Hash mapping collision using MD5 is very small. However, since it takes 16 bytes of storage space to store one URL, it requires a lot of memory space when storing billions of URL information.

• 2) In-memory database based storage

To store a huge number of URL libraries, the method of in-memory storage faces the problem of memory overflow. In fact, we can use external memory to assist in storage. In-memory databases based on Key-Value storage model Berkeley DB and Redis are both famous No SQL databases, which are very suitable for URL repositories in web crawlers because of their simple data storage model and support for high concurrency and random storage. The problem of URL de-duplication can be easily solved by using the compressed URL byte array as the key and the corresponding value with a Boolean value to identify the crawl state.

• 3) Bloom Filter-based storage

Bloom Filter is a space-efficient and fast random data structure that maps each string to an n-bit binary vector using k Hash functions. The Bloom filter can quickly determine whether an element is already in the set and only uses the storage space of an n-dimensional array, which is the optimal choice for URL de-duplication methods in large-scale web crawlers.

6.3.2 Data Acquisition in the State of Missing Field Information

• (1) Specific system interface data collection methods

As for a specific data source website, the web pages of the website often have a similar structure, and by discovering templates and generating data wrappers, accurate opinion information extraction can be achieved. As for microblog and forum data sources, on the one hand, the number of such data sites to be collected in this paper is small but the layout of each other's pages is very different; on the other hand, the data of these data sources are organized in a fixed format and the data are related to each other, such as the data of the index page of the main post of the forum and the data of the content page of the main post, and the data of the blog posts and comments of microblog, which require accurate data extraction to ensure the correct operation of the collection program and thus extract the correct and formatted data of public opinion information. Therefore, the rule template design method is used here, and the data layout templates are manually marked for different microblogging and forum data sources and added to the rule template library for subsequent information extraction. The general flow of web information extraction based on rule template design is shown in Fig. 6.22.

Fig. 6.22

Rule-based template approach to web information extraction

Specific operating instructions are as follows:

• 1) As for a specific data source, first open the web page source code through Google chrome browser's developer tools, locate the specific topic information area, such as microblog user information, then record the CSS path of the topic information and add it to the specific topic rule template for that data source;

• 2) When parsing the downloaded pages, we will look for the corresponding rule templates from the rule template library and then match the corresponding theme information CSS paths one by one;

• 3) If the rule matches successfully, use the JSoup tool to extract the corresponding topic information;

Nowadays, HTML web pages are generally designed by the DIV+CCS layout method. CSS is short for Cascading Style Sheets, a technique used to express the style of documents such as HTML or XML. JSoup is a simple, fast and efficient java web parsing tool that provides a very easy API to extract and manipulate data through DOM, CSS and j Query-like operations. In practical engineering practice, JSoup+CSSPath technology is used to build microblog and BBS forum web information extraction rule templates.

• (2) Other data collection methods on the external network

Distributed crawlers are further divided into master–slave distributed and peer-to-peer distributed crawlers according to the crawling model. Master–slave distributed crawler has a dedicated Master node to maintain the queue of URLs to be crawled and is responsible for distributing URLs to different Slave nodes each time, while the Slave nodes are responsible for the actual data collection and sending the extracted URL links to the Master node, where URL de-duplication is done. In addition to maintaining the queue of URLs to be crawled, the Master node also balances the load of each Slave by distributing URLs. The shortcoming of this model is the performance bottleneck of the Master node. Today's fairly mature distributed crawler architecture is the Python-based Scrapy-Redis crawler architecture, where the Scrapy framework loads web page downloading and parsing, and the distributed key-value memory database Redis builds URL queues, which solves the performance bottleneck of the Master node. In peer-to-peer distributed web crawler mode, all nodes of the cluster share the same work, and each crawler server can get URLs from the queue of URLs to be crawled, and then the URL host name or domain name is hash-processed and distributed to the corresponding crawler nodes after the hash value is modulo the number of machine nodes. The early shortcoming of this model is its scalability, as a machine downtime leads to redistribution of crawler tasks, resulting in a waste of resources.

Here we use the first distributed data collection method and build a URL queue using a Redis database. The architecture of the distributed web news page collection design is shown in Fig. 6.23. The Master node acts as the control node and uses the Redis in-memory database to manage the URL crawler queue, including the storage of URL links and the de-duplication of URLs. Here we use the same number of URL queues in Redis based on the number of Slave nodes to store the URL links of the corresponding web news sites, so that each Slave node can retrieve the URL links from the corresponding URL queue in Redis. Here the URL de-duplication is done with the help of a global Bloom filter, that is, before the Spider parser on the Slave node extracts a new URL link and puts it into the Redis crawler queue, it is checked by the Bloom filter first, if the URL is not in the Bloom filter, then it is added to the Bloom filter and inserted into the corresponding URL queue, otherwise, the URL link is discarded.

Fig. 6.23

Distributed web page collection architecture design

The Slave node runs a web crawler program that obtains URL links from the Master node URL repository, downloads web pages, parses web news content and stores it in the HBase database, then extracts the URL links and returns them to the Master node. The workflow of the Spider program run by the Slave node is shown in Fig. 6.24.

Fig. 6.24

Workflow diagram of Spider on Slave node

• 1) start, initializing configuration parameters, including connection to the master node URL repository;

• 2) Based on the manually assigned set of news gathering sites, Spider's task scheduler fetches the URL links with crawling from the corresponding Redis queue on the Master node

• 3) Web page downloader to download web news pages;

• 4) The web page parser on the one hand retrieves information such as web news body, title and release time, and stores it in HBase database after formatting, and on the other hand extracts news external links from the web page;

• 5) regular matching of the extracted URL link to ensure that the link is a page from the assigned site collection and then returned to the master node via the scheduler;

• 6) The URL links returned to the master node are first de-duplicated by the global Bloom Filter and then added to the corresponding Redis queue.

Considering that web news sites will update a large number of messages every day, in order to achieve the function of incremental collection of web news data, we need to regularly inject artificial seed URLs, which are to collect links to the home pages of certain sections of news sites, because these home page information will be dynamically updated every day. Based on these seed URLs, you can constantly extract the newly updated web news links every day to achieve the function of incremental collection of web news data.

• (3) Satellite transmission mode information acquisition

Some power disaster data is not available from existing systems or internal data, and first-hand data needs to be obtained from power disaster sites. However, sometimes geological disasters or floods occur on site, which can destroy the real-time communication on site and make the primary data collected on site unable to be transmitted back to the system in time. At this time, it is necessary to consider using on-site information acquisition satellite communication module for satellite communication, transmitting the first-hand data collected in the field, back to the system in a timely manner through satellite transmission. As shown in Fig. 6.25.

Fig. 6.25

Field information acquisition module communication system

The on-site information collection module based on satellite communication has 2G, 3G and 4G communication functions at the same time. In the areas with good ground public network or wireless network, the on-site information collection module can transmit the collected information back to the system in real-time through the public network or wireless network, and because it involves intranet data security, a security interaction platform is specially set up. When the information collection site is hit by serious natural disasters, such as mudslides, earthquakes, floods, etc., the natural disasters will cause destruction to the ground public network, and then the information collected through the site information collection module cannot be transmitted back to the emergency command center in time, which requires the satellite communication function of the emergency terminal to be activated to transmit the collected site first-hand power disaster data back to the emergency command center through satellite in time for reference by emergency commanders.

6.3.3 Data Fusion Processing of Comprehensive Social Source Information

• (1) Principle of multi-source information fusion

Multi-source information fusion is a fundamental function prevalent in human or other biological systems. Humans understand their surroundings and ongoing events by applying this ability to combine information (external objects, sounds, smells, touch) from various sensors (eyes, ears, nose, limbs) of the human body and use a priori knowledge to count them. The basic principle of multi-sensor information fusion technology is like the human brain integrated processing information, making full use of multiple sensor resources and combining the redundant or complementary information of multiple sensors in time and space according to some criteria through the rational domination and use of these sensors and their observation information, so as to obtain a consistent interpretation or description of the observed object.

Data fusion system also means that various real-time, non-real-time, accurate, fuzzy, fast-varying, gradual-varying, similar or contradictory data from various data sources are rationally allocated and used, and the redundant or complementary information is synthesized and analyzed according to some specific rules, so as to obtain a comprehensive description of the measured object.

Specifically, the principle of multi-source data fusion is as follows:

• 1) Multiple types of data collection on the target;

• 2) Feature extraction of the collected data and extraction of the feature vector representing the target measurement data

• 3) Effective pattern recognition processing of feature vectors using artificial intelligence or other methods such as pattern recognition that can convert feature vectors of targets into attribute judgments to complete the description of each sensor data about the target under test;

• 4) Grouping of the description data of the objectives in association with the same objective, based on the results of the previous step;

• 5) The grouped data by each target is synthesized using an appropriate fusion algorithm to obtain a more accurate and consistent interpretation and description of the target under test.

• (2) Data fusion scheme for integrated social source information in the absence of on-site information

The characteristics of power disaster data make it necessary for data fusion to calculate each row and column of the data, completing both the correspondence of each column parameter of different data sources and the matching between different data sources for each row, using an attribute-based matching method. The main idea of the designed algorithm is to discover the columns and rows with high similarity in the data and use them as the seed data, and then use them as the starting point to learn to discover other new matching columns and rows until the matching of the whole data is completed. This has the advantage of locking the column matching to the most probable columns, reducing the computational effort of matching all columns and finding the global optimal solution through the local optimal solution. The specific algorithm design block diagram is shown in Fig. 6.26.

Fig. 6.26

Block diagram of data fusion method

The block diagram of the improved data fusion method is shown above. By analyzing the data, setting the seed features → finding matches from the seed features → calculating the similarity and updating the similarity table → getting a new match table → …… This cycle is carried out, and finally the matching of all data is completed to get the result of data fusion. The improved algorithm has the following advantages by matching the potentially better matches in the data: 1) Being rewindable with high accuracy; 2) no need to annotate data, which can be applied to multi-source data fusion; 3) Fast convergence and small number of iterations; 4) Data fusion of the matched parameters to complement the data and improve the confidence level; 5) Low complexity of calculation; 6) All kinds of parameters are adjustable and highly flexible.

The seed feature columns, i.e., the columns selected at the outset to best represent the data characteristics, need to have low overlap in this data source and use a fairly homogeneous description specification across data sources. Since it serves as the beginning of the matching process and affects the subsequent matching process, the selection of a good seed feature column can speed up the subsequent matching process while helping to find a globally optimal solution, so the selection of the seed feature column is crucial. Two parameters are proposed to measure and select the seed feature columns. One is the degree of repetition of the parameter columns and the other is the degree of descriptive uniformity of the parameter columns from different data sources. Repetition is used to measure the degree of repetition of the data description of a parameter column of a single data source, i.e., the degree of difference between parameters using that parameter column, with a greater degree of difference indicating a greater likelihood that it can be used as a unique distinguishing identifier. The higher the degree of uniformity, the higher the standard of description of the same thing in both data sources, and thus the higher the usability of using it as a matching criterion. Feature columns require low repetition and high descriptive uniformity, and also need to be selected as seed features by considering the actual characteristics of the commodity data.

After determining the seed feature columns, it is necessary to calculate the matching degree between different data sources, which is obtained by weighting the similarity of each attribute and is calculated using the attribute based similarity measure method. The attribute-based similarity measure is a measure of similarity using the weighted sum of similarity between individual attribute values.

Definition of similarity: Suppose a message has multiple attributes. As for a single attribute, let it be able to represent its similarity to another message with a number from 0 to 1, the higher the value means, the higher the similarity. For the similarity measure of attributes, different similarity calculation methods can be selected according to the specific data types of attributes, and then different weights can be assigned to different attribute values according to the a priori knowledge and the actual situation of the data to characterize their different importance in entity unification, and finally the similarity of each attribute is weighted and summed to obtain the similarity between entities, and two entities with similarity greater than a certain threshold are considered to be similar.

Specific calculations: Calculation of the similarity of genus between ${c}_{i}$, ${d}_{j}$, marked as ${sim}_{attr}\left({c}_{i},{d}_{j}\right)$. ${sim}_{attr}\left({c}_{i},{d}_{j}\right)={\sum }_{k}{sim}_{attr}\left({c}_{i}\left(k\right),{d}_{j}\left(k\right)\right)*{P}_{k}$, where ${sim}_{attr}\left({c}_{i}\left(k\right),{d}_{j}\left(k\right)\right)$ denotes the similarity between ${c}_{i}$, ${d}_{j}$ matching attributes k, ${P}_{k}$ represents the weight of attribute K.

Similarity calculation: There are several methods for calculating the similarity of character type data based on semantic similarity, statistical association and literal similarity. The semantic similarity method is achieved by understanding the semantic meaning of character strings, and strings with the same semantic meaning are considered the same; The statistical correlation party method calculates the similarity of strings based on their statistically similar data; the literal similarity method is mainly based on the edit distance and synonym (word) methods. Among them, the edit distance is widely used because of its fairly mature calculation method and fairly good results.

The edit distance is used to measure the similarity between the attributes of the character type, and the multiple of deviation from the standard deviation is used to measure the similarity between the attributes of numeric type. The edit distance is the number of times two strings become the same character by the least number of edit operations (insert, replace, delete) between them. The edit distance mainly measures the errors that may occur in the string input transmission on a character-by-character basis. The larger edit distance indicates the greater difference between two strings, so the smaller the similarity, and the smaller the edit distance, the greater the similarity. As for data-valued data, the edit distance method does not apply to measuring the similarity because its values have specific meanings. In this chapter, the standard deviation of the attribute column of the value type is calculated in advance, and the difference between two attribute values is measured by the multiple of the standard deviation. In general, the larger the value, the smaller the similarity, and if the value is 0, the two are equal.

Standardization of similarity: Since the edit distance and the multiple of standard deviation are both unrestricted in the data range, and since different attributes need to be weighted to get a total similarity, the edit distance of individual attributes cannot be too large to affect the similarity of the whole entity too much, while the edit distances of different attributes also need to be standardized to make all the data have the same range. Both the edit distance and the difference standard deviation multiplier take values in the range (0, +∞), and in practice, they are generally finite. In order to facilitate understanding, we need to map them into a uniform fixed finite interval, and the commonly used normalization methods are min–max normalization and z-score (standard score) normalization. The min–max normalization is more influenced by the max–min values, but can ensure that all values are within a certain range. Z-score is not sensitive to the max–min values and can reflect the characteristics of the overall values, but the range of values is uncertain. In order to avoid the large influence of the maximum and minimum values on the data, the z-score normalization is used here to normalize the above distances. The specific steps are as follows:

Step 1: Find the initial matching row based on the seed feature column. By having set the seed feature column ${A}_{k1}$ of data source A, seed feature column ${B}_{k1}$ of data source B (can be other columns, specified here for illustrative purposes), calculate the similarity ${sim}_{attr}\left({a}_{i1},{b}_{j1}\right)$ of each entity of data sources A and B with respect to the seed column (i takes 0–n, j takes 0–m), entities whose similarity reaches a set threshold are considered similar at this stage. Here it is assumed that the similar entities obtained are ${A}_{c1}$ and ${B}_{c1}$, ${A}_{c2}$ and${B}_{c2}$.

Step 2: Find a new matching column based on the initial matching row. Based on the similarity pairs obtained, the similarity between individual attributes is calculated, and the average similarity of known pairs of attributes in each column is calculated. When the similarity of an attribute column is higher than a set threshold, the column is considered similar. Therefore, calculate ${A}_{c1}$ and ${B}_{c1}$, each of the attributes ${sim}_{attr}\left({a}_{1i},{b}_{1j}\right)$ and ${sim}_{attr}\left({a}_{2i},{b}_{2j}\right)$ of ${A}_{c2}$ and ${B}_{c2}$(i takes 0–n, j takes 0–m), and calculate the average similarity between each column pair ${sim}_{attr}\left({A}_{ki},{B}_{kj}\right)=avg\left({sim}_{attr}\left({a}_{vi},{b}_{wj}\right)\right)$ (where v, w are paired entity pairs), as ${sim}_{attr}\left({A}_{ki},{B}_{kj}\right)=avg\left({sim}_{attr}\left({a}_{1i},{b}_{1j}\right)+{sim}_{attr}\left({a}_{2i},{b}_{2j}\right)\right)$. Assuming that here we get similar attribute column pairs as ${A}_{k2}$, ${B}_{k2}$.

Step 3: Find new matching rows based on the new matching columns. Based on the newly obtained feature column pairs, calculate the similarity of each entity of each data source on these attributes, get the new similarity between each entity, and update the similar entity pairs table according to the threshold value. This process involves both new entity pairs being judged as new matching entity pairs and entity pairs that have already been judged as matches being judged as non-matching entity pairs. This allows backtracking results to ensure that the matching results find a locally optimal solution based on the added evidence, and use this locally optimal solution to approximate the global optimal solution.

Step 4: Based on the newly obtained matched entity pairs, calculate the similarity between the attributes of each entity pair, and calculate the similarity of each attribute column to the matched entity pair to get the new matched attribute column. The overall process is similar to step 3, but with a different entry angle. The difference is that when matching rows by columns, each attribute has a different weight, while when matching columns by rows, each row has the same weight, and when matching columns, column names (parameter names) are also involved in the matching process. If any matching rows change, proceed to step 3; if not, proceed to step 5.

Step 5: Data conflict resolution for the matched data. The matched rows (entities) are obtained from step 4, where the attribute values of the matched attributes will have the possibility of data conflict, and data conflict resolution is needed to get more credible and complete data, which is one of the purposes of data fusion.

6.3.4 Power Emergency Warning and Response Process Command Decision

• (1) Electricity emergency warning assisted decision-making

Comprehensive social source information, early warning for power facilities, auxiliary power emergency warning, early access to affected power facilities, early warning response.

The power emergency warning assisted decision-making process is as follows, and the flow chart is shown in Fig. 6.27.

Fig. 6.27

Schematic diagram of the power emergency warning aid decision model

Step 1: Obtaining information from social sources and processing the information to get disaster warning information;

Step 2: Combining the disaster warning information with the grid GIS data information to obtain the affected grid facilities through data analysis and processing;

Step 3: Conducting disaster warning analysis and research based on the affected grid facilities, further analyzing the degree of impact of the disaster on the grid, and thus deriving recommendations for warning issuance;

Step 4: Issuing an internal warning for early warning response after analysis and discussion are confirmed;

Step 5: Obtaining personnel, team, material, and equipment data information to aid in the deployment of early warning response;

Step 6: Real-time follow-up and update of disaster warning information, re-analysis and research, and real-time update of early warning response decisions based on personnel, team, material, and equipment data information to assist in power emergency warning work.

• (2) Electricity emergency disaster damage prediction model

The prediction model of power event development trend (loss of power equipment and facilities) is shown in Fig. 6.28.

Fig. 6.28

Schematic diagram of the event trend prediction model

The execution process of power emergency trend prediction is shown in the figure.

Step 1: Based on the underlying and current data, the forecast data can be used for damage prediction based on the proposed power emergency scenario projection;

Step 2: Obtaining scenario extrapolation of disaster development through basic data and other data, and obtaining specific predicted data on damage to electrical equipment and facilities through scenario extrapolation;

Step 3: The content of the scenario extrapolation is mainly based on historical damage records, and the specific data are corrected by the macro disaster loss prediction results in order to obtain more accurate trend prediction results;

Step 4: The event development prediction data is influenced by the emergency disposal command decision, so it is necessary to continuously track the emergency disposal process, and update the results of the event development prediction in a timely manner while constantly updating the internal data and external data to ensure the real-time nature of the prediction results.

The prediction process is shown in Fig. 6.29.

Fig. 6.29

Outage range prediction model

Step 1: Based on the relevant basic data, current damage data can be obtained for the current state of the outage range and the results of the prediction of the event trend;

Step 2: Based on the results of the event development trend prediction, the results of the outage scope prediction can be obtained by analyzing the impact capacity of the damage status of the power facilities in the results. It is necessary to realize the results of determining the depth of the analysis and gradually analyzing the secondary impact and tertiary impact of power facility damage.

Step 3: The number of outage areas and outage users in the forecast results can assist in the prediction of outage scope in two ways. The number of outage users and the number of station areas can provide more powerful prediction data for speculating the size of the outage range. At the same time, the prediction results of outage users, outage station areas, and outage important users can be used to reverse the prediction of which power equipment is more likely to be damaged.

Step 4: As the event develops, the outage scope prediction results are continuously updated based on the changes in the data.

• (3) Command center and field information interaction model

In the absence of information at the emergency site, the interaction model between the command center and the information on site is established to realize the transmission of information at the disaster site and the communication of command center instructions, as shown in Fig. 6.30.

Fig. 6.30

Command center and field information interaction model

The command center interacts with the field information model in the following steps:

Step 1: In-depth survey and information gathering through a base team;

Step 2: The base team transmits the collected on-site information, including on-site video, on-site pictures, text descriptions, voice information, etc., through the mobile public network or satellite communication;

Step 3: After receiving the relevant information, the on-site command will analyze and judge the situation, mobilize the repair team nearby to carry out repairs, and at the same time make applications for material allocation to ensure the supply of emergency materials;

Step 4: The on-site command reports the relevant information to the emergency command center, or the base team reports the information directly to the emergency command center by means of mobile public network communication or satellite communication;

Step 5: After receiving the information, the emergency command center will analyze and research, and can transmit the instructions to the field command, or directly to the nearest material deployment;

Step 6: The emergency command center can also conduct cross-region material deployment for cross-region material support;

Step 7: The emergency command center can directly convey instructions to the repair team, which carries out continuous repair work at the disaster site.

Realize real-time interaction between emergency command center and field information through information transmission from base team, repair team and field command.

• (4) Interaction model between resource deployment auxiliary decision-making command center and on-site information

The generation of resource deployment-assisted decision scenarios can be refined on the basis of the forecast results:

Input resource reassessment: An important issue is that the results of input application at that time may not be optimal, which means that more maintenance resources can be dispatched from other areas that are not affected to improve maintenance efficiency in order to achieve the best maintenance results. At this level the data on the amount of resource input needs to be increased appropriately according to the actual analysis results.

Resource statistics based on emergency capability pool instances: Based on the specific work content of emergency capability pool instances, the actual amount of resources required during the execution of each capability is counted. At this stage, the specific level of implementation of each competency in the competency pool is derived, that is, how many times each competency can be implemented or how long it can be sustained.

Task Bank-Capability Bank Correspondence Calculation: Based on the results of disaster damage prediction and disaster development trend prediction, correspondence adjustment calculation is performed with the emergency capability bank instance, and each task in the task bank instance is determined to determine whether it has the ability to execute the response plan in the capability bank.

Specific redeployment program design: Based on the number of emergency resources and the time required for maintenance, a specific resource redeployment program is developed. Various factors need to be considered such as the distance of emergency resources from the failure site, continuous working time, consumables management, and energy management. Also consider the change in the scope of the area where the outage occurs and develop a final and reasonable emergency resource dispatch plan.