A dynamic programming segmentation procedure for hydrological and environmental time series

Ath. Kehagias; Ev. Nidelkou; V. Petridis

doi:10.1007/s00477-005-0013-6

Stochastic Environmental Research and Risk Assessment

January 2006, Volume 20, Issue 1–2, pp 77–94 | Cite as

A dynamic programming segmentation procedure for hydrological and environmental time series

Authors
Authors and affiliations

Ath. KehagiasEmail author
Ev. Nidelkou
V. Petridis

Original Paper

First Online: 08 October 2005

209 Downloads
29 Citations

Abstract

We present a procedure for the segmentation of hydrological and environmental time series. The procedure is based on the minimization of Hubert’s segmentation cost or various generalizations of this cost. This is achieved through a dynamic programming algorithm, which is guaranteed to find the globally optimal segmentations with K=1, 2, ..., K _max segments. Various enhancements can be used to speed up the basic dynamic programming algorithm, for example recursive computation of segment errors and “block segmentation”. The “true” value of K is selected through the use of the Bayesian information criterion. We evaluate the segmentation procedure with experiments which involve artificial as well as temperature and river discharge time series.

Keywords

Time series Segmentation Change point Dynamic programming River discharge

This is a preview of subscription content, to check access.

Appendix 1

Recursive formulas for segment error

In this appendix we prove the validity of the recursive formulas (Eqs. 8, 9, 10, 11, 12) for the computation of segment error. Recall that Q _{s, t}=U _{s, t} ^′ U _{s, t},R _{s, t}=U _{s, t} ^′ X _{s, t}. We have

$${\bf Q}_{s, t+1}={\bf U}_{s, t+1}^{\prime}{\bf U}_{s, t+1}=\left[ \begin{array}{*{20}c}{\bf U}_{s, t}^{\prime}& | & {\bf u}_{t+1}^{\prime}\end{array}\right] \left[\begin{array}{*{20}c}{\bf U}_{s, t}^{\prime}\\ --\\ u_{t+1} \end{array}\right] ={\bf U}_{s, t}^{\prime}{\bf U}_{s, t}+{\bf u}_{t+1}^{\prime }{\bf u}_{t+1}={\bf Q}_{s, t}+{\bf u}_{t+1}^{\prime}{\bf u}_{t+1}. $$

(16)

Similarly, for R _{s, t} we obtain

$${\bf R}_{s, t+1}={\bf R}_{s, t}+{\bf u}_{t+1}^{\prime}x_{t+1}. $$

(17)

Furthermore, we have d _{s, t}=| X _{s, t}−U _{s, t} A _{s, t}| ² and

$$\begin{aligned} d_{s, t+1}& =\left\vert {\bf X}_{s, t+1}-{\bf U}_{s, t+1}{\bf A} _{s, t+1}\right\vert ^{2}\\ & = \left\vert {\bf X}_{s, t}-{\bf U}_{s, t}{\bf A}_{s, t+1}\right\vert ^{2}+\left\vert x_{t+1}-{\bf u}_{t+1}{\bf A}_{s, t+1}\right\vert ^{2}. \\ \end{aligned} $$

(18)

Now

$$\begin{aligned} \left\vert {\bf X}_{s, t}-{\bf U}_{s, t}{\bf A}_{s, t+1}\right\vert ^{2} & =\left\vert {\bf X}_{s, t}-{\bf U}_{s, t}{\bf A} _{s, t}+{\bf U}_{s, t}{\bf A}_{s, t}-{\bf U}_{s, t}{\bf A} _{s, t+1}\right\vert ^{2} \\ & = \left \vert {\bf X}_{s, t}-{\bf U}_{s, t}{\bf A}_{s, t}\right\vert ^{2}+\left\vert {\bf U}_{s, t}\left( {\bf A}_{s, t}-{\bf A} _{s, t+1}\right) \right\vert ^{2}+2\left( {\bf X}_{s, t}-{\bf U} _{s, t}{\bf A}_{s, t}\right) ^{\prime}{\bf U}_{s, t}\left( {\bf A} _{s, t}-{\bf A}_{s, t+1}\right) \end{aligned} $$

Setting

$${\bf \delta A}={\bf A}_{s, t}-{\bf A}_{s, t+1}, $$

(19)

we have

$$\left\vert {\bf X}_{s, t}-{\bf U}_{s, t}{\bf A}_{s, t+1}\right\vert ^{2}=\left\vert {\bf X}_{s, t}-{\bf U}_{s, t}{\bf A}_{s, t}\right\vert ^{2}+\left\vert {\bf U}_{s, t}{\bf \delta A}\right\vert ^{2}+2\left( {\bf X}_{s, t}-{\bf U}_{s, t}{\bf A}_{s, t}\right) ^{\prime} {\bf U}_{s, t}{\bf \delta A}. $$

(20)

Also, we have

$$d_{s, t}=\left\vert {\bf X}_{s, t}-{\bf U}_{s, t}{\bf A}_{s, t} \right\vert ^{2}; $$

(21)

and

$$\left\vert {\bf U}_{s, t}{\bf \delta A}\right\vert ^{2}=\left( {\bf U}_{s, t}{\bf \delta A}\right) ^{\prime}{\bf U}_{s, t} {\bf \delta A}={\bf \delta A}^{\prime}{\bf U}_{s, t}^{\prime }{\bf U}_{s, t}{\bf \delta A}={\bf \delta A}^{\prime}{\bf Q} _{s, t}{\bf \delta A;} $$

(22)

and

$$\begin{aligned} \left( {\bf X}_{s, t}-{\bf U}_{s, t}{\bf A}_{s, t}\right)^{\prime }{\bf U}_{s, t}{\bf \delta A} & = \left( {\bf U}_{s, t}^{\prime}{\bf X}_{s, t}-{\bf U}_{s, t}^{\prime}{\bf U}_{s, t}{\bf A} _{s, t}\right)^{\prime}{\bf \delta A}\\ & = \left( {\bf U}_{s, t}^{\prime}{\bf X}_{s, t}-{\bf U}_{s, t} ^{\prime}{\bf U}_{s, t}\left( {\bf U}_{s, t}^{\prime}{\bf U} _{s, t}\right) ^{-1}{\bf U}_{s, t}^{\prime}{\bf X}_{s, t}\right) ^{\prime}{\bf \delta A}\\ & = 0. \\ \end{aligned} $$

(23)

Replacing Eqs. 21, 22, 23 in Eq. 20 we get

$$\left\vert {\bf X}_{s, t}-{\bf U}_{s, t}{\bf A}_{s, t+1}\right\vert ^{2}=d_{s, t}+{\bf \delta A}^{\prime}{\bf Q}_{s, t}{\bf \delta A} $$

(24)

and replacing Eq. 24 in Eq. 18 we get

$$d_{s, t+1}=d_{s, t}+{\bf \delta A}^{\prime}{\bf Q}_{s, t}{\bf \delta A}+\left\vert x_{t+1}-{\bf u}_{t+1}{\bf A}_{s, t+1}\right\vert ^{2}. $$

(25)

Equations 16, 17, 19, 25 constitute the recursion presented in Sect. 3.2. The initialization Q _s,s=u _s ^′ u _s, R _{s, s}=u _s ^′ x _s, A _{s, s}= Q _s,s ⁻¹ R _{s, s} follows from the definition of Q _{s, t}, R _{s, t}, A _{s, t}.

Appendix 2

Block segmentation

In this appendix we present the modifications necessary for block segmentation. In block segmentation we assume that segment boundaries can only appear at constant multiples of a number N, which we call block length. For example, if T=1,000 and N=10, the only candidate boundaries are 0, 10, 20, ..., 1,000; this reduces computation by a factor of 100, at the cost of obtaining a “coarse” segmentation.

The definition of d _{s, t} must be modified; now it is the error of segment [(s−1)·N+1, t·N]. As far as computation is concerned, it is only required to change the input and output regression variables; u _t remains unchanged, but we now define

$${\bf X}_{s, t} =\left[\begin{array}{*{20}c} x_{\left( s-1\right) \cdot N+1}\\ x_{\left( s-1\right) \cdot N+2}\\ \ldots\\ x_{t\cdot N} \end{array}\right], \quad \widehat{{\bf X}}_{s, t} =\left[\begin{array}{*{20}c}\widehat{x}_{\left( s-1\right) \cdot N+1}\\ \widehat{x}_{\left( s-1\right)\cdot N+2}\\ \ldots\\ \widehat{x}_{t\cdot N} \end{array}\right], \quad{\bf U}_{s, t}=\left[\begin{array}{*{20}c}{\bf u}_{\left( s-1\right) \cdot N+1+1}\\ {\bf u}_{\left(s-1\right) \cdot N+1+2}\\ \ldots\\ {\bf u}_{t\cdot N}\end{array}\right]; $$

(26)

compare with Eq. 6. The formulas for d _{s, t} and A _{s, t} also remain unchanged:

$$ d_{s, t}=\left( {\bf X}_{s, t}{\bf -U}_{s, t}{\bf A}_{s, t}\right) ^{\prime}\left( {\bf X}_{s, t}{\bf -U}_{s, t}{\bf A}_{s, t}\right) ,\quad{\bf A}_{s, t}=\left( {\bf U}_{s, t}^{\prime}{\bf U} _{s, t}\right) ^{-1}{\bf U}_{s, t}^{\prime}{\bf X}_{s, t}. $$

Finally, the recursive formula for d _{s, t} is modified as follows

$$d_{s, t+1}=d_{s, t}+{\bf \delta A}^{\prime}{\bf Q}_{s, t}{\bf \delta A}+\left\vert {\bf X}_{t, t+1}-{\bf U}_{t, t+1}{\bf A}_{s, t+1} \right\vert ^{2}. $$

(27)

The segmentation algorithm of Sect. 3.2 remains unchanged, but now uses as input the newly defined block segment error d _{s, t}.

Appendix 3

Beeferman’s segmentation metric P _k

Beeferman’s segmentation metric P _k(s, t) measures the error of a proposed segmentation s=(0,s ₁ , ..., s _K-1, t) with respect to a “true” segmentation t=(0, t ₁ , ..., t _L-1, t). P _k first appeared in the text segmentation literature (Beeferman et al. 1999) but it can be applied to any segmentation problem. Here we will only give a short intuitive discussion of P _k. The interested reader can find more details in (Beeferman et al. 1999).

When is s identical to t? The following two conditions are necessary and sufficient.

1.

Each pair of integers (i, j) which is in the same segment under t is also in the same segment under s.
2.

Each pair (i, j) in a different segment under t is also in a different segment under s.

It follows that the difference between s and t (i.e. the error of s with respect to t) is the amount by which the above criteria are violated. This can happen in two ways:

1.

Some pair (i, j) which is in the same segment under t, is in a different segment under s;
2.

Or some pair (i, j) which is in the same segment under t, is in a different segment under s.

We can formalize the above description as follows. Define a function δ_s(i, j) to be 1 when i and j are in the same segment under s and 0 otherwise; define δ_t (i, j) similarly. Then we are interested in the quantity

$$\sum_{i=1}^{T}\sum_{j=1}^{T}\left\vert \delta_{{\bf s}}\left( i, j\right) -\delta_{{\bf t}}\left( i, j\right) \right\vert . $$

(28)

There are two problems with Eq. 28 and they both have to do with the range of the summations. First, considering all possible pairs (i, j) is too time consuming. Second, it yields an undiscriminating criterion, because even a grossly wrong s will agree with t on many pairs (for example most pairs (i, i+1) will be placed in the same segment and most (i, j) pairs which are very far apart will be placed in different segments). Beeferman et al. propose to consider only pairs which are k steps apart (i, i+k+1), where k is half the average segment length. It has been empirically validated that this choice of k works well; in (Beeferman et al. 1999) Beeferman et al. also discuss some theoretical justification for this choice. Hence P _k is defined as follows

$$P_{k}\left( {\bf s, t}\right) =\sum_{i=1}^{T-k-1}\left\vert \delta _{{\bf s}}\left( i, i+k+1\right) -\delta_{{\bf t}}\left( i, i +k+1\right) \right\vert $$

(29)

and this is what we have used in Sect. 4.

References

Akaike H (1974) A new look at the statistical model identification. IEEE Trans Automat Control 19:716–723CrossRefGoogle Scholar
Auger IE, Lawrence CE (1989) Algorithms for the optimal identification of segment neighborhoods. Bull Math Biol 51:39–54PubMedGoogle Scholar
Basseville M, Nikiforov IV (1993) Detection of abrupt changes—theory and application. Prentice-Hall, Englewood CliffsGoogle Scholar
Beeferman D, Berger A, Lafferty J (1999) Statistical models for text segmentation. Machine learning 34:177–210CrossRefGoogle Scholar
Bellman R (1961) On the approximation of curves by line segments using dynamic programming. Commun ACM 4:284CrossRefGoogle Scholar
Braun JV, Mueller H-G (1998) Statistical methods for DNA sequence segmentation. Stat Sci 13:142–162CrossRefGoogle Scholar
Braun JV, Braun RK, Mueller H-G (2000) Multiple changepoint fitting via quasilikelihood, with application to DNA sequence segmentation. Biometrika 87:301–314CrossRefGoogle Scholar
Chen H-L (2002) Testing hydrologic time series for stationarity. J Hydr Eng 7:129–136CrossRefGoogle Scholar
Chen J, Gupta AK (1998) Information criterion and change point problem for regular models. Tech. Rep. No. 98-05, Department of Math. and Stat., Bowling Green State U., OhioGoogle Scholar
Fortin V, Perreault L, Salas JD (2004) Retrospective analysis and forecasting of streamflows using a shifting level model. J Hydrol 296:135–163CrossRefGoogle Scholar
Fortin V, Perreault L, Salas JD (2004) Analyse retrospective et prevision des debits en presence de changement de regime. 57e Congres Annuel de l’ Association Canadienne des Ressources HydriquesGoogle Scholar
Himberg J, Korpiaho K, Mannila H, Tikanmaki J, Toivonen HTT (2001) Time series segmentation for context recognition in mobile devices. Proc. of ICDM 2001, pp 203–210Google Scholar
Hipel KW, McLeod AI (1994) Time series modelling of water resources and environmental systems. Elsevier, AmsterdamCrossRefGoogle Scholar
Hubert P (1997) Change points in meteorological analysis. In: Subba Rao T, Priestley MB, Lessi O (eds) Applications of time series analysis in astronomy and meteorology. Chapman and Hall, LondonGoogle Scholar
Hubert P (2000) The segmentation procedure as a tool for discrete modeling of hydrometeorogical regimes. Stoch Env Res Risk Ass 14:297–304CrossRefGoogle Scholar
Jackson B et al (2005) An algorithm for optimal partitioning of data on an interval. IEEE Signal Proc Lett 12:105–108CrossRefGoogle Scholar
Kay SM (1998) Fundamentals of statistical signal processing: detection theory. Prentice-Hall, Englewood CliffsGoogle Scholar
Kearns M, Mansour Y, Ng AY, Ron D (1997) An experimental and theoretical comparison of model selection methods. Machine Learn 27:7–50CrossRefGoogle Scholar
Kehagias Ath (2004) A hidden Markov model segmentation procedure for hydrological and environmental time series. Stoch Environ Res Risk Assess 18:117–130CrossRefGoogle Scholar
Keogh E et al (2001) An online algorithm for segmenting time series. In: Proceedings of the 2001 IEEE International Conference on Data Mining, pp 289–296Google Scholar
Keogh E, Chu S, Hart D, Pazzani M (2003) Segmenting time series: a survey and novel approach, In: Data mining in time series databases. World Scientific Publishing Company, SingaporeGoogle Scholar
Mann ME, Bradley RS, Hughes MK (1999) Northern hemisphere temperatures during the past millennium: inferences, uncertainties, and limitations. Geophys Res Lett 26:759–762CrossRefGoogle Scholar
Ninomiya Y (2003) Asymptotic theory for change-point model and its application to model selection. Tech. Report, Graduate School of Mathematics, Kyusho Univ., JapanGoogle Scholar
Pavlidis T, Horowitz SL (1974) Segmentation of Plane Curves. IEEE Trans Comput 23:860–870CrossRefGoogle Scholar
Schwarz G (1978) Estimating the dimension of a model. Ann Stat 6:461–464CrossRefGoogle Scholar
Sveinsson OGB, Salas JD, Boes DC, Pielke RA Sr (2003) Modeling the dynamics of long term variability of hydroclimatic processes. J Hydrometeorol 4:489–505CrossRefGoogle Scholar
Yao Y-C (1988) Estimating the number of change-points via Schwarz’ criterion. Stat Prob Lett 6:181–189CrossRefGoogle Scholar

Copyright information

Authors and Affiliations

Ath. Kehagias
- 1
Email author
Ev. Nidelkou
- 1
V. Petridis
- 1

1.Faculty of EngineeringAristotle University of ThessalonikiThessalonikiGreece

A dynamic programming segmentation procedure for hydrological and environmental time series

Abstract

Keywords

References

Copyright information

Authors and Affiliations

Personalised recommendations

Cite article