Temporal Scale Selection in Time-Causal Scale Space

When designing and developing scale selection mechanisms for generating hypotheses about characteristic scales in signals, it is essential that the selected scale levels reflect the extent of the underlying structures in the signal. This paper presents a theory and in-depth theoretical analysis about the scale selection properties of methods for automatically selecting local temporal scales in time-dependent signals based on local extrema over temporal scales of scale-normalized temporal derivative responses. Specifically, this paper develops a novel theoretical framework for performing such temporal scale selection over a time-causal and time-recursive temporal domain as is necessary when processing continuous video or audio streams in real time or when modelling biological perception. For a recently developed time-causal and time-recursive scale-space concept defined by convolution with a scale-invariant limit kernel, we show that it is possible to transfer a large number of the desirable scale selection properties that hold for the Gaussian scale-space concept over a non-causal temporal domain to this temporal scale-space concept over a truly time-causal domain. Specifically, we show that for this temporal scale-space concept, it is possible to achieve true temporal scale invariance although the temporal scale levels have to be discrete, which is a novel theoretical construction. The analysis starts from a detailed comparison of different temporal scale-space concepts and their relative advantages and disadvantages, leading the focus to a class of recently extended time-causal and time-recursive temporal scale-space concepts based on first-order integrators or equivalently truncated exponential kernels coupled in cascade. Specifically, by the discrete nature of the temporal scale levels in this class of time-causal scale-space concepts, we study two special cases of distributing the intermediate temporal scale levels, by using either a uniform distribution in terms of the variance of the composed temporal scale-space kernel or a logarithmic distribution. In the case of a uniform distribution of the temporal scale levels, we show that scale selection based on local extrema of scale-normalized derivatives over temporal scales makes it possible to estimate the temporal duration of sparse local features defined in terms of temporal extrema of first- or second-order temporal derivative responses. For dense features modelled as a sine wave, the lack of temporal scale invariance does, however, constitute a major limitation for handling dense temporal structures of different temporal duration in a uniform manner. In the case of a logarithmic distribution of the temporal scale levels, specifically taken to the limit of a time-causal limit kernel with an infinitely dense distribution of the temporal scale levels towards zero temporal scale, we show that it is possible to achieve true temporal scale invariance to handle dense features modelled as a sine wave in a uniform manner over different temporal durations of the temporal structures as well to achieve more general temporal scale invariance for any signal over any temporal scaling transformation with a scaling factor that is an integer power of the distribution parameter of the time-causal limit kernel. It is shown how these temporal scale selection properties developed for a pure temporal domain carry over to feature detectors defined over time-causal spatio-temporal and spectro-temporal domains.

Abstract When designing and developing scale selection mechanisms for generating hypotheses about characteristic scales in signals, it is essential that the selected scale levels reflect the extent of the underlying structures in the signal.
This paper presents a theory and in-depth theoretical analysis about the scale selection properties of methods for automatically selecting local temporal scales in time-dependent signals based on local extrema over temporal scales of scalenormalized temporal derivative responses. Specifically, this paper develops a novel theoretical framework for performing such temporal scale selection over a time-causal and time-recursive temporal domain as is necessary when processing continuous video or audio streams in real time or when modelling biological perception.
For a recently developed time-causal and time-recursive scale-space concept defined by convolution with a scaleinvariant limit kernel, we show that it is possible to transfer a large number of the desirable scale selection properties that hold for the Gaussian scale-space concept over a non-causal temporal domain to this temporal scale-space concept over a truly time-causal domain. Specifically, we show that for this temporal scale-space concept, it is possible to achieve true temporal scale invariance although the temporal scale levels have to be discrete, which is a novel theoretical construction.
The analysis starts from a detailed comparison of different temporal scale-space concepts and their relative advantages and disadvantages, leading the focus to a class of recently extended time-causal and time-recursive temporal scale-space concepts based on first-order integrators or equiv-  100 44 Stockholm, Sweden. E-mail: tony@kth.se alently truncated exponential kernels coupled in cascade. Specifically, by the discrete nature of the temporal scale levels in this class of time-causal scale-space concepts, we study two special cases of distributing the intermediate temporal scale levels, by using either a uniform distribution in terms of the variance of the composed temporal scale-space kernel or a logarithmic distribution.
In the case of a uniform distribution of the temporal scale levels, we show that scale selection based on local extrema of scale-normalized derivatives over temporal scales makes it possible to estimate the temporal duration of sparse local features defined in terms of temporal extrema of first-or second-order temporal derivative responses. For dense features modelled as a sine wave, the lack of temporal scale invariance does, however, constitute a major limitation for handling dense temporal structures of different temporal duration in a uniform manner.
In the case of a logarithmic distribution of the temporal scale levels, specifically taken to the limit of a timecausal limit kernel with an infinitely dense distribution of the temporal scale levels towards zero temporal scale, we show that it is possible to achieve true temporal scale invariance to handle dense features modelled as a sine wave in a uniform manner over different temporal durations of the temporal structures as well to achieve more general temporal scale invariance for any signal over any temporal scaling transformation with a scaling factor that is an integer power of the distribution parameter of the time-causal limit kernel.
It is shown how these temporal scale selection properties developed for a pure temporal domain carry over to fea-

Introduction
When processing sensory data by automatic methods in areas of signal processing such as computer vision or audio processing or in computational modelling of biological perception, the notion of receptive field constitutes an essential concept (Hubel and Wiesel [33,34]; Aertsen and Johannesma [2]; DeAngelis et al. [16,15]; Miller et al [94]).
For sensory data as obtained from vision or hearing, or their counterparts in artificial perception, the measurement from a single light sensor in a video camera or on the retina, or the instantaneous sound pressure registered by a microphone is hardly meaningful at all, since any such measurement is strongly dependent on external factors such as the illumination of a visual scene regarding vision or the distance between the sound source and the microphone regarding hearing. Instead, the essential information is carried by the relative relations between local measurements at different points and temporal moments regarding vision or local measurements over different frequencies and temporal moments regarding hearing. Following this paradigm, sensory measurements should be performed over local neighbourhoods over space-time regarding vision and over local neighbourhoods in the time-frequency domain regarding hearing, leading to the notions of spatio-temporal and spectro-temporal receptive fields.
A general problem when applying the notion of receptive fields in practice, however, is that the types of responses that are obtained in a specific situation can be strongly dependent on the scale levels at which they are computed. A spatio-temporal receptive field is determined by at least a spatial scale parameter and a temporal scale parameter, whereas a spectro-temporal receptive field is determined by at least a spectral and a temporal scale parameter. Beyond ensuring that local sensory measurements at different spatial, temporal and spectral scales are treated in a consistent manner, which by itself provides strong contraints on the shapes of the receptive fields (Lindeberg [72,78]; Lindeberg and Friberg [82,83]), it is necessary for computer vision or machine hearing algorithms to decide what responses within the families of receptive fields over different spatial, temporal and spectral scales they should base their analysis on.
Over the spatial domain, theoretically well-founded methods have been developed for choosing spatial scale levels among receptive field responses over multiple spatial scales (Lindeberg [66,65,68,74,75]) leading to e.g. robust methods for image-based matching and recognition (Lowe [89]; Mikolajczyk and Schmid [92]; Tuytelaars and van Gool [116]; Bay et al. [5]; Tuytelaars and Mikolajczyk [117]; van de Sande et al. [105]; Larsen et al. [52]) that are able to handle large variations of the size of the objects in the image domain and with numerous applications regarding object recognition, object categorization, multi-view geometry, construction of 3-D models from visual input, human-computer interaction, biometrics and robotics.
Much less research has, however, been performed regarding the topic of choosing local appropriate scales in temporal data. While some methods for temporal scale selection have been developed (Lindeberg [64]; Laptev and Lindeberg [50]; Willems et al. [121]), these methods suffer from either theoretical or practical limitations.
A main subject of this paper is present a theory for how to compare filter responses in terms of temporal derivatives that have been computed at different temporal scales, specifically with a detailed theoretical analysis of the possibilities of having temporal scale estimates as obtained from a temporal scale selection mechanism reflect the temporal duration of the underlying temporal structures that gave rise to the feature responses. Another main subject of this paper is to present a theoretical framework for temporal scale selection that leads to temporal scale invariance and enables the computation of scale covariant temporal scale estimates. While these topics can for a non-causal temporal domain be addressed by the non-causal Gaussian scale-space concept (Iijima [35]; Witkin [122]; Koenderink [45]; Koenderink and van Doorn [47]; Lindeberg [61,62,70]; Florack [22]; ter Haar Romeny [26]), the development of such a theory has been missing regarding a time-causal temporal domain.

Temporal scale selection
When processing time-dependent signals in video or audio or more generally any temporal signal, special attention has to be put to the facts that: the physical phenomena that generate the temporal signals may occur at different speed -faster or slower, and the temporal signals may contain qualitatively different types of temporal structures at different temporal scales.
In certain controlled situations where the physical system that generates the temporal signals that is to be processed is sufficiently well known and if the variability of the temporal scales over time in the domain is sufficiently constrained, suitable temporal scales for processing the signals may in some situations be chosen manually and then be verified experimentally. If the sources that generate the temporal signals are sufficiently complex and/or if the temporal structures in the signals vary substantially in temporal duration by the underlying physical processes occurring significantly faster or slower, it is on the other hand natural to (i) include a mechanism for processing the temporal data at multiple temporal scales and (ii) try to detect in a bottom-up manner at what temporal scales the interesting temporal phenomena are likely to occur. The subject of this article is to develop a theory for temporal scale selection in a time-causal temporal scale space as an extension of a previously developed theory for spatial scale selection in a spatial scale space (Lindeberg [66,65,68,74,75]), to generate bottom-up hypotheses about characteristic temporal scales in time-dependent signals, intended to serve as estimates of the temporal duration of local temporal structures in time-dependent signals. Special focus will be on developing mechanisms analogous to scale selection in non-causal Gaussian scale-space, based on local extrema over scales of scale-normalized derivatives, while expressed within the framework of a time-causal and time-recursive temporal scale space in which the future cannot be accessed and the signal processing operations are thereby only allowed to make use of information from the present moment and a compact buffer of what has occurred in the past.
When designing and developing such scale selection mechanisms, it is essential that the computed scale estimates reflect the temporal duration of the corresponding temporal structures that gave rise to the feature responses. To understand the pre-requisites for developing such temporal scale selection methods, we will in this paper perform an in-depth theoretical analysis of the scale selection properties that such temporal scale selection mechanisms give rise to for different temporal scale-space concepts and for different ways of defining scale-normalized temporal derivatives.
Specifically, after an examination of the theoretical properties of different types of temporal scale-space concepts, we will focus on a class of recently extended time-causal temporal scale-space concepts obtained by convolution with truncated exponential kernels coupled in cascade (Lindeberg [57,77,78]; Lindeberg and Fagerström [81]). For two natural ways of distributing the discrete temporal scale levels in such a representation, in terms of either a uniform distribution over the scale parameter τ corresponding to the variance of the composed scale-space kernel or a logarithmic distribution, we will study the scale selection properties that result from detecting local temporal scale levels from local extrema over scale of scale-normalized temporal derivatives. The motivation for studying a logarithmic distribution of the temporal scale levels, is that it corresponds to a uniform distribution in units of effective scale τ ef f = A + B log τ for some constants A and B, which has been shown to consti-tute the natural metric for measuring the scale levels in a spatial scale space (Koenderink [45]; Lindeberg [59]).
As we shall see from the detailed theoretical analysis that will follow, this will imply certain differences in scale selection properties of a temporally asymmetric time-causal scale space compared to scale selection in a spatially mirror symmetric Gaussian scale space. These differences in theoretical properties are in turn essential to take into explicit account when formulating algorithms for temporal scale selection in e.g. video analysis or audio analysis applications.
For the temporal scale-space concept based on a uniform distribution of the temporal scale levels in units of the variance of the composed scale-space kernel, it will be shown that temporal scale selection from local extrema over temporal scales will make it possible to estimate the temporal duration of local temporal structures modelled as local temporal peaks and local temporal ramps. For a dense temporal structure modelled as a temporal sine wave, the lack of true scale invariance for this concept will, however, imply that the temporal scale estimates will not be directly proportional to the wavelength of the temporal sine wave. Instead, the scale estimates are affected by a bias, which is not a desirable property.
For the temporal scale-space concept based on a logarithmic distribution of the temporal scale levels, and taken to the limit to scale-invariant time-causal limit kernel (Lindeberg [78]) corresponding to an infinite number of temporal scale levels that cluster infinitely close near the temporal scale level zero, it will on the other hand be shown that the temporal scale estimates of a dense temporal sine wave will be truly proportional to the wavelength of the signal. By a general proof, it will be shown this scale invariant property of temporal scale estimates can also be extended to any sufficiently regular signal, which constitutes a general foundation for expressing scale invariant temporal scale selection mechanisms for time-dependent video and audio and more generally also other classes of time-dependent measurement signals.
As complement to this proposed overall framework for temporal scale selection, we will also present a set of general theoretical results regarding time-causal scale-space representations: (i) showing that previous application of the assumption of a semi-group property for time-causal scalespace concepts leads to undesirable temporal dynamics, which however can be remedied by replacing the assumption of a semi-group structure be a weaker assumption of a cascade property in turn based on a transitivity property, (ii) formulations of scale-normalized temporal derivatives for Koenderink's time-causal scale-time model [46] and (iii) ways of translating the temporal scale estimates from local extrema over temporal scales in the temporal scale-space representation based on the scale-invariant time-causal limit kernel into quantitative measures of the temporal duration of the corresponding underlying temporal structures and in turn based on a scale-time approximation of the limit kernel.
In these ways, this paper is intended to provide a theoretical foundation for expressing theoretically well-founded temporal scale selection methods for selecting local temporal scales over time-causal temporal domains, such as video and audio with specific focus on real-time image or sound streams. Applications of this scale selection methodology for detecting both sparse and dense spatio-temporal features in video are presented in a companion paper [79].

Structure of this article
As a conceptual background to the theoretical developments that will be performed, we will start in Section 2 with an overview of different approaches to handling temporal data within the scale-space framework including a comparison of relative advantages and disadvantages of different types of temporal scale-space concepts.
As a theoretical baseline for the later developments of methods for temporal scale selection in a time-causal scale space, we shall then in Section 3 give an overall description of basic temporal scale selection properties that will hold if the non-causal Gaussian scale-space concept with its corresponding selection methodology for a spatial image domain is applied to a one-dimensional non-causal temporal domain, e.g. for the purpose of handling the temporal domain when analysing pre-recorded video or audio in an offline setting.
In Sections 4-5 we will then continue with a theoretical analysis of the consequences of performing temporal scale selection in the time-causal scale space obtained by convolution with truncated exponential kernels coupled in cascade (Lindeberg [57,77,78]; Lindeberg and Fagerström [81]). By selecting local temporal scales from the scales at which scale-normalized temporal derivatives assume local extrema over temporal scales, we will analyze the resulting temporal scale selection properties for two ways of defining scale-normalized temporal derivatives, by either variancebased normalization as determined by a scale normalization parameter γ or L p -normalization for different values of the scale normalization power p.
With the temporal scale levels required to be discrete because of the very nature of this temporal scale-space concept, we will specifically study two ways of distributing the temporal scale levels over scale, using either a uniform distribution relative to the temporal scale parameter τ corresponding to the variance of the composed temporal scalespace kernel in Section 4 or a logarithmic distribution of the temporal scale levels in Section 5.
Because of the analytically simpler form for the timecausal scale-space kernels corresponding to a uniform distri-bution of the temporal scale levels, some theoretical scalespace properties will turn out to be easier to study in closed form for this temporal scale-space concept. We will specifically show that for a temporal peak modelled as the impulse response to a set of truncated exponential kernels coupled in cascade, the selected temporal scale level will serve as a good approximation of the temporal duration of the peak or be proportional to this measure depending on the value of the scale normalization parameter γ used for scalenormalized temporal derivatives based on variance-based normalization or the scale normalization power p for scale-normalized temporal derivatives based on L p -normalization. For a temporal onset ramp, the selected temporal scale level will on the other hand be either a good approximation of the time constant of the onset ramp or proportional to this measure of the temporal duration of the ramp. For a temporal sine wave, the selected temporal scale level will, however, not be directly proportional to the wavelength of the signal, but instead affected by a systematic bias. Furthermore, the corresponding scale-normalized magnitude measures will not be independent of the wavelength of the sine wave but instead show systematic wavelength dependent deviations. A main reason for this is that this temporal scale-space concept does not guarantee temporal scale invariance if the temporal scale levels are distributed uniformly in terms of the temporal scale parameter τ corresponding to the temporal variance of the temporal scale-space kernel.
With a logarithmic distribution of the temporal scale levels, we will on the other hand show that for the temporal scale-space concept defined by convolution with the timecausal limit kernel (Lindeberg [78]) corresponding to an infinitely dense distribution of the temporal scale levels towards zero temporal scale, the temporal scale estimates will be perfectly proportional to the wavelength of a sine wave for this temporal scale-space concept. It will also be shown that this temporal scale-space concept leads to perfect scale invariance in the sense that (i) local extrema over temporal scales are preserved under temporal scaling factors corresponding to integer powers of the distribution parameter c of the time-causal limit kernel underlying this temporal scalespace concept and are transformed in a scale-covariant way for any temporal input signal and (ii) if the scale normalization parameter γ = 1 or equivalently if the scale normalization power p = 1, the magnitude values at the local extrema over scale will be equal under corresponding temporal scaling transformations. For this temporal scale-space concept we can therefore fulfil basic requirements to achieve temporal scale invariance also over a time-causal and timerecursive temporal domain.
To simplify the theoretical analysis we will in some cases temporarily extend the definitions of temporal scale-space representations over discrete temporal scale levels to a continuous scale variable, to make it possible to compute local extrema over temporal scales from differentiation with respect to the temporal scale parameter. Section 6 discusses the influence that this approximation has on the overall theoretical analysis.
Section 7 then illustrates how the proposed theory for temporal scale selection can be used for computing local scale estimates from 1-D signals with substantial variabilities in the characteristic temporal duration of the underlying structures in the temporal signal.
In Section 8, we analyse how the derived scale selection properties carry over to a set of spatio-temporal feature detectors defined over both multiple spatial scales and multiple temporal scales in a time-causal spatio-temporal scale-space representation for video analysis. Section 9 then outlines how corresponding selection of local temporal and logspectral scales can be expressed for audio analysis operations over a time-causal spectro-temporal domain. Finally, Section 10 concludes with a summary and discussion.
To simplify the presentation, we have put some derivations and theoretical analysis in the appendix. Appendix A presents a general theoretical argument of why a requirement about a semi-group property over temporal scales will lead to undesirable temporal dynamics for a time-causal scale space and argue that the essential structure of non-creation of new image structures from any finer to any coarser temporal scale can instead nevertheless be achieved with the less restrictive assumption about a cascade smoothing property over temporal scales, which then allows for better temporal dynamics in terms of e.g. shorter temporal delays.
In relation to Koenderink's scale-time model [46], Appendix B shows how corresponding notions of scale-normalized temporal derivatives based on either variance-based normalization or L p -normalization can be defined also for this time-causal temporal scale-space concept.
Appendix C shows how the temporal duration of the time-causal limit kernel proposed in (Lindeberg [78]) can be estimated by a scale-time approximation of the limit kernel via Koenderink's scale-time model leading to estimates of how a selected temporal scale levelτ from local extrema over temporal scale can be translated into a estimates of the temporal duration of temporal structures in the temporal scale-space representation obtained by convolution with the time-causal limit kernel. Specifically, explicit expressions are given for such temporal duration estimates based on first-and second-order temporal derivatives.
2 Theoretical background and related work

Temporal scale-space concepts
For processing temporal signals at multiple temporal scales, different types of temporal scale-space concepts have been developed in the computer vision literature (see Figure 1): For off-line processing of pre-recorded signals, a noncausal Gaussian temporal scale-space concept may in many situations be sufficient. A Gaussian temporal scale-space concept is constructed over the 1-D temporal domain in a similar manner as a Gaussian spatial scale-space concept is constructed over a D-dimensional spatial domain (Iijima [35]; Witkin [122]; Koenderink [45]; Koenderink and van Doorn [47]; Lindeberg [61,62,70]; Florack [22]; ter Haar Romeny [26]), with or without the difference that a model for temporal delays may or may not be additionally included (Lindeberg [70]).
When processing temporal signals in real time, or when modelling sensory processes in biological perception computationally, it is on the other hand necessary to base the temporal analysis on time-causal operations.
The first time-causal temporal scale-space concept was developed by Koenderink [46], who proposed to apply Gaussian smoothing on a logarithmically transformed time axis with the present moment mapped to the unreachable infinity. This temporal scale-space concept does, however, not have any known time-recursive formulation. Formally, it requires an infinite memory of the past and has therefore not been extensively applied in computational applications.
Lindeberg [57,77,78] and Lindeberg and Fagerström [81] proposed a time-causal temporal scale-space concept based on truncated exponential kernels or equivalently first-order integrators coupled in cascade, based on theoretical results by Schoenberg [108] (see also Schoenberg [109] and Karlin [42]) implying that such kernels are the only variationdiminishing kernels over a 1-D temporal domain that guarantee non-creation of new local extrema or equivalently zerocrossings with increasing temporal scale. This temporal scalespace concept is additionally time-recursive and can be implemented in terms of computationally highly efficient firstorder integrators or recursive filters over time. This theory has been recently extended into a scale-invariant time-causal limit kernel (Lindeberg [78]), which allows for scale invariance over the temporal scaling transformations that correspond to exact mappings between the temporal scale levels in the temporal scale-space representation based on a discrete set of logarithmically distributed temporal scale levels.
Based on semi-groups that guarantee either self-similarity over temporal scales or non-enhancement of local extrema with increasing temporal scales, Fagerström [20] and Lindeberg [70] have derived time-causal semi-groups that allow for a continuous temporal scale parameter and studied theoretical properties of these kernels.
Concerning temporal processing over discrete time, Fleet and Langley [21] performed temporal filtering for optic flow computations based on recursive filters over time. Lindeberg [57,77,78] and Lindeberg and Fagerström [81] showed that first-order recursive filters coupled in cascade constitutes a natural time-causal scale-space concept over discrete time, h(t; µ, K = 10) ht(t; µ, K = 10) htt(t; µ, K = 10) Fig. 1 Temporal scale-space kernels with composed temporal variance τ = 1 for the main types of temporal scale-space concepts considered in this paper and with their first-and second-order temporal derivatives: (top row) the non-causal Gaussian kernel g(t; τ ), (second row) the composition h(t; µ, K = 10) of K = 10 truncated exponential kernels with equal time constants, (third row) the composition h(t; K = 10, c = √ 2) of K = 10 truncated exponential kernels with logarithmic distribution of the temporal scale levels for c = √ 2, (fourth row) corresponding kernels h(t; K = 10, c = 2) for c = 2, (fifth row) Koenderink's scale-time kernels h Koe (t; c = √ 2) corresponding to Gaussian convolution over a logarithmically transformed temporal axis with the parameters determined to match the time-causal limit kernel corresponding to truncated exponential kernels with an infinite number of logarithmically distributed temporal scale levels according to (186) for c = √ 2, (bottom row) corresponding scale-time kernels h Koe (t; c = 2) for c = 2. (Horizontal axis: time t) based on the requirement that the temporal filtering over a 1-D temporal signal must not increase the number of local extrema or equivalently the number of zero-crossings in the signal. In the specific case when all the time constants in this model are equal and tend to zero while simultaneously increasing the number of temporal smoothing steps in such a way that the composed temporal variance is held constant, these kernels can be shown to approach the temporal Poisson kernel [81]. If on the other hand the time constants of the first-order integrators are chosen so that the temporal scale levels become logarithmically distributed, these temporal smoothing kernels approach a discrete approximation of the time-causal limit kernel [78].
Applications of the temporal scale-space model based on truncated exponential kernels with equal time constants coupled in cascade and corresponding to Laguerre functions (Laguerre polynomials multiplied by a truncated exponential kernel) for computing spatio-temporal features have presented by Rivero-Moreno and Bres [103], Shabani et al. [111] and Berg et al. [6] as well as for handling time scales in video surveillance (Jacob and Pless [37]), for performing edge preserving smoothing in video streams (Paris [98]) and is closely related to Tikhonov regularization as used for image restoration by e.g. Surya et al. [115]. A general framework for performing spatio-temporal feature detection based on the temporal scale-space model based on truncated exponential kernels coupled in cascade with specifically the both theoretical and practical advantages of using logarithmic distribution of the intermediated temporal scale levels in terms of temporal scale invariance and better temporal dynamics (shorter temporal delays) has been presented in Lindeberg [78].

Relative advantages of different temporal scale spaces
When developing a temporal scale selection mechanism over a time-causal temporal domain, a first problem concerns what time-causal scale-space concept to base the multi-scale temporal analysis upon. The above reviewed temporal scalespace concepts have different relative advantages from a theoretical and computational viewpoint. In this section, we will perform an in-depth examination of the different temporal scale-space concepts that have been developed in the literature, which will lead us to a class of time-causal scalespace concepts that we argue is particularly suitable with respect to the set of desirable properties we aim at.
The non-causal Gaussian temporal scale space is in many cases the conceptually easiest temporal scale-space concept to handle and to study analytically (Lindeberg [70]). The corresponding temporal kernels are scale invariant, have compact closed-form expressions over both the temporal and frequency domains and obey a semi-group property over temporal scales. When applied to pre-recorded signals, temporal delays can if desirable be disregarded, which eliminates any need for temporal delay compensation. This scale-space concept is, however, not time-causal and not time-recursive, which implies fundamental limitations with regard to realtime applications and realistic modelling of biological perception.
Koenderink's scale-time kernels [46] are truly time-causal, allow for a continuous temporal scale parameter, have good temporal dynamics and have a compact explicit expression over the temporal domain. These kernels are, however, not time-recursive, which implies that they in principle require an infinite memory of the past (or at least extended temporal buffers corresponding to the temporal extent to which the infinite support temporal kernels are truncated at the tail). Thereby, the application of Koenderink's scale-time model to video analysis implies that substantial temporal buffers are needed when implementing this non-recursive temporal scale-space in practice. Similar problems with substantial need for extended temporal buffers arise when applying the non-causal Gaussian temporal scale-space concept to offline analysis of extended video sequences. The algebraic expressions for the temporal kernels in the scale-time model are furthermore not always straightforward to handle and there is no known simple expression for the Fourier transform of these kernels or no known simple explicit cascade smoothing property over temporal scales with respect to the regular (untransformed) temporal domain. Thereby, certain algebraic calculations with the scale-time kernels may become quite complicated.
The temporal scale-space kernels obtained by coupling truncated exponential kernels or equivalently first-order integrators in cascade are both truly time-causal and truly timerecursive (Lindeberg [57,77,78]; Lindeberg and Fagerström [81]). The temporal scale levels are on the other hand required to be discrete. If the goal is to construct a real-time signal processing system that analyses continuous streams of signal data in real time, one can however argue that a restriction of the theory to a discrete set of temporal scale levels is less of a contraint, since the signal processing system anyway has to be based on a finite amount of sensors and hardware/wetware for sampling and processing the continuous stream of signal data.
In the special case when all the time constants are equal, the corresponding temporal kernels in the temporal scalespace model based on truncated exponential kernels coupled in cascade have compact explicit expressions that are easy to handle both in the temporal domain and in the frequency domain, which simplifies theoretical analysis. These kernels obey a semi-group property over temporal scales, but are not scale invariant and lead to slower temporal dynamics when a larger number of primitive temporal filters are coupled in cascade (Lindeberg [77,78]).
In the special case when the temporal scale levels in this scale-space model are logarithmically distributed, these kernels have a manageable explicit expression over the Fourier domain that enables some closed-form theoretical calculations. Deriving an explicit expression over the temporal domain is, however, harder, since the explicit expression then corresponds to a linear combination of truncated exponential filters for all the time constants, with the coefficients determined from a partial fraction expansion of the Fourier transform, which may lead to rather complex closed-form expressions. Thereby certain analytical calculations may become harder to handle. As shown in [78] and Appendix C, some such calculations can on the other hand be well approximated via a scale-time approximation of the time-causal temporal scale-space kernels. When using a logarithmic distribution of the temporal scales, the composed temporal kernels do however have very good temporal dynamics and much better temporal dynamics compared to corresponding kernels obtained by using truncated exponential kernels with equal time constants coupled in cascade. Moreover, these kernels lead to a computationally very efficient numerical implementation. Specifically, these kernels allow for the formulation of a time-causal limit kernel that obeys scale invariance under temporal scaling transformations, which cannot be achieved if using a uniform distribution of the temporal scale levels (Lindeberg [77,78]).
The temporal scale-space representations obtained from the self-similar time-causal semi-groups have a continuous scale parameter and obey temporal scale invariance (Fagerström [20]; Lindeberg [70]). These kernels do, however, have less desirable temporal dynamics (see Appendix A for a general theoretical argument about undesirable consequences of imposing a temporal semi-group property on temporal kernels with temporal delays) and/or lead to pseudodifferential equations that are harder to handle both theoretically and in terms of computational implementation. For these reasons, we shall not consider those time-causal semi-groups further in this treatment.

Previous work on methods for scale selection
A general framework for performing scale selection for local differential operations was proposed in Lindeberg [60,61] based on the detection of local extrema over scale of scalenormalized derivative expressions and then refined in Lindeberg [66,65] -see Lindeberg [68,75] for tutorial overviews.
Alternative approaches for performing scale selection over spatial image domains have also been proposed in terms of (i) detecting peaks of weighted entropy measures (Kadir and Brady [40]) or Lyaponov functionals (Sporring et al. [114]) over scales, (ii) minimising normalized error measures over scale (Lindeberg [67]), (iii) determining minimum reliable scales for edge detection based on a noise suppression model (Elder and Zucker [18]), (iv) determining at what scale levels to stop in non-linear diffusion-based image restoration methods based on similarity measurements relative to the original image data (Mrázek and Navara [95]), (v) by comparing reliability measures from statistical classifiers for texture analysis at multiple scales (Kang et al. [41]), (vi) by computing image segmentations from the scales at which a supervised classifier delivers class labels with the highest reliability measure (Loog et al. [88]; Li et al. [56]), (vii) selecting scales for edge detection by estimating the saliency of elongated edge segments (Liu et al. [87]) or (viii) considering subspaces generated by local image descriptors computed over multiple scales (Hassner et al. [31]).
More generally, spatial scale selection can be seen as a specific instance of computing invariant receptive field responses under natural image transformations, to (i) handle objects in the world of different physical size and to account for scaling transformations caused by the perspective mapping, and with extensions to (ii) affine image deformations to account for variations in the viewing direction and (iii) Galilean transformations to account for relative motions between objects in the world and the observer as well as to (iv) illumination variations (Lindeberg [73]).
Early theoretical work on temporal scale selection in a time-causal scale space was presented in Lindeberg [64] with primary focus on the temporal Poisson scale-space, which possesses a temporal semi-group structure over a discrete time-causal temporal domain while leading to long temporal delays (see Appendix A for a general theoretical argument). Temporal scale selection in non-causal Gaussian spatio-temporal scale space has been used by Laptev and Lindeberg [50] and Willems et al. [121] for computing spatio-temporal interest points, however, with certain theoretical limitations that are explained in a companion paper [79]. 1 The purpose of this article is to present a much further developed and more general theory for temporal scale selection in timecausal scale spaces over continuous temporal domains and 1 The spatio-temporal scale selection method in (Laptev and Lindeberg [50]) is based on a spatio-temporal Laplacian operator that is not scale covariant under independent relative scaling transformations of the spatial vs. the temporal domains [79], which implies that the spatial and temporal scale estimate will not be robust under independent variabilities of the spatial and temporal scales in video data. The spatio-temporal scale selection method applied to the determinant of the spatio-temporal Hessian in (Willems et al. [121]) does not make use of the full flexibility of the notion of γ-normalized derivative operators [79] and has not previously been developed over a time-causal spatio-temporal domain.
to analyse the theoretical scale selection properties for different types of model signals.
3 Scale selection properties for the non-causal Gaussian temporal scale space concept In this section, we will present an overview of theoretical properties that will hold if the Gaussian temporal scale-space concept is applied to a non-causal temporal domain, if additionally the scale selection mechanism that has been developed for a non-causal spatial domain is directly transferred to a non-causal temporal domain. The set of temporal scalespace properties that we will arrive at will then be used as a theoretical base-line for developing temporal scale-space properties over a time-causal temporal domain.

Non-causal Gaussian temporal scale-space
Over a one-dimensional temporal domain, axiomatic derivations of a temporal scale-space representation based on the assumptions of (i) linearity, (ii) temporal shift invariance, (iii) semi-group property over temporal scale, (iv) sufficient regularity properties over time and temporal scale and (v) nonenhancement of local extrema imply that the temporal scalespace representation L(·; τ, δ) = g(·; τ, δ) * f (·) (1) should be generated by convolution with possibly time-delayed temporal kernels of the form (Lindeberg [70]) where τ is a temporal scale parameter corresponding to the variance of the Gaussian kernel and δ is a temporal delay. Differentiating the kernel with respect to time gives see the top row in Figure 1 for graphs. When analyzing prerecorded temporal signals, it can be preferable to set the temporal delay to zero, leading to temporal scale-space kernels having a similar form as spatial Gaussian kernels:

Temporal scale selection from scale-normalized derivatives
As a conceptual background to the treatments that we shall later develop regarding temporal scale selection in time-causal temporal scale spaces, we will in this section describe the theoretical structure that arises by transferring the theory for scale selection in a Gaussian scale space over a spatial domain to the non-causal Gaussian temporal scale space: Given the temporal scale-space representation L(t; τ ) of a temporal signal f (t) obtained by convolution with the Gaussian kernel g(t; τ ) according to (1), temporal scale selection can be performed by detecting local extrema over temporal scales of differential expressions expressed in terms of scale-normalized temporal derivatives at any scale τ according to (Lindeberg [66,65,68,75]) where ζ = t/τ γ/2 is the scale-normalized temporal variable, n is the order of temporal differentiation and γ is a free parameter. It can be shown [66, Section 9.1] that this notion of γ-normalized derivatives corresponds to normalizing the nth order Gaussian derivatives g ζ n (t; τ ) over a one-dimensional domain to constant L p -norms over scale τ with where the perfectly scale invariant case γ = 1 corresponds to L 1 -normalization for all orders n of temporal differentiation.
Temporal scale invariance. A general and very useful scale invariant property that results from this construction of the notion of scale-normalized temporal derivatives can be stated as follows: Consider two signals f and f that are related by a temporal scaling transformation and assume that there is a local extremum over scales at (t 0 ; τ 0 ) in a differential expression D γ−norm L defined as a homogeneous polynomial of Gaussian derivatives computed from the scale-space representation L of the original signal f . Then, there will be a corresponding local extremum over scales at (t 0 ; τ 0 ) = (S t 0 ; S 2 τ 0 ) in the corresponding differential expression D γ−norm L computed from the scale-space representation L of the rescaled signal f [66, Section 4.1]. This scaling result holds for all homogeneous polynomial differential expression and implies that local extrema over scales of γ-normalized derivatives are preserved under scaling transformations. Specifically, this scale invariant property implies that if a local scale temporal level level in dimension of time σ = τ is selected to be proportional to the temporal scale estimateσ = √τ such that σ = Cσ, then if the temporal signal f is transformed by a temporal scale factor S, the temporal scale estimate and therefore also the selected temporal scale level will be transformed by a similar temporal factorσ = Sσ, implying that the selected temporal scale levels will automatically adapt to variations in the characteristic temporal scale of the signal. Thereby, such local extrema over temporal scale provide a theoretically wellfounded way to automatically adapt the scale levels to local scale variations.
Specifically, scale-normalized scale-space derivatives of order n at corresponding temporal moments will be related according to which means that γ = 1 implies perfect scale-invariance in the sense that the γ-normalized derivatives at corresponding points will be equal. If γ = 1, the difference in magnitude can on the other hand be easily compensated for using the scale values of the corresponding scale-adaptive image features (see below).

Temporal peak
For a temporal peak modelled as a Gaussian function with variance τ 0 it can be shown that scale selection from local extrema over scale of second-order scale-normalized temporal derivatives implies that the scale estimate at the position t = 0 of the peak will be given by (Lindeberg [65,Equation (56)] [74, If we require the scale estimate to reflect the temporal duration of the peak such that then this implies which in the specific case of q = 1 corresponds to [65, Section 5.6.1] and in turn corresponding to L p -normalization for p = 2/3 according to (8).
If we additionally renormalize the original Gaussian peak to having maximum value equal to one then if using the same value of γ for computing the magnitude response as for selecting the temporal scale, the maximum magnitude value over scales will be given by and will not be independent of the temporal scale τ 0 of the original peak unless γ = 1. If on the other hand using γ = 3/4 as motivated by requirements of scale calibration (14) for q = 1, the scale dependency will for a Gaussian peak be of the form To get a scale-invariant magnitude measure for comparing the responses of second-order temporal derivative responses at different temporal scales for the purpose of scale calibration, we should therefore consider a scale-invariant magnitude measure for peak detection of the form which for a Gaussian temporal peak will assume the value Specifically, this form of post-normalization corresponds to computing the scale-normalized derivatives for γ = 1 at the selected scale (14) of the temporal peak, which according to (8) corresponds to L 1 -normalization of the second-order temporal derivative kernels.

Temporal onset ramp
If we model a temporal onset ramp with temporal duration τ 0 as the primitive function of the Gaussian kernel with variance τ 0 it can be shown that scale selection from local extrema over scale of first-order scale-normalized temporal derivatives implies that the scale estimate at the central position t = 0 will be given by [65, Equation (23)] If we require this scale estimate to reflect the temporal duration of the ramp such that then this implies which in the specific case of q = 1 corresponds to [65, Section 4.5.1] and in turn corresponding to L p -normalization for p = 2/3 according to (8). If using the same value of γ for computing the magnitude response as for selecting the temporal scale, the maximum magnitude value over scales will be given by which is not independent of the temporal scale τ 0 of the original onset ramp unless γ = 1. If using γ = 1 for temporal scale selection, the selected temporal scale according to (24) would, however, become infinite. If on the other hand using γ = 1/2 as motivated by requirements of scale calibration (25) for q = 1, the scale dependency will for a Gaussian onset ramp be of the form To get a scale-invariant magnitude measure for comparing the responses of first-order temporal derivative responses at different temporal scales, we should therefore consider a scale-invariant magnitude measure for ramp detection of the form which for a Gaussian onset ramp will assume the value Specifically, this form of post-normalization corresponds to computing the scale-normalized derivatives for γ = 1 at the selected scale (25) of the onset ramp and thus also to L pnormalization of the first-order temporal derivative kernels for p = 1.

Temporal sine wave
For a signal defined as a temporal sine wave it can be shown that there will be a peak over temporal scales in the magnitude of the nth order temporal derivative L ζ n = τ nγ/2 L t n at temporal scale [66, Section 3] If we define a temporal scale parameter σ of dimension [time] according to σ = √ τ , then this implies that the scale estimate is proportional to the wavelength λ 0 = 2π/ω 0 of the sine wave according to [66,Equation (9)] and does in this respect reflect a characteristic time constant over which the temporal phenomena occur. Specifically, the maximum magnitude measure over scale [66, Equation (10)] is for γ = 1 independent of the angular frequency ω 0 of the sine wave and thereby scale invariant.
In the following, we shall investigate how these scale selection properties can be transferred to two types of timecausal temporal scale-space concepts.

Scale selection properties for the time-causal temporal scale space concept based on first-order integrators with equal time constants
In this section, we will present a theoretical analysis of the scale selection properties that are obtained in the time-causal scale-space based on truncated exponential kernels coupled in cascade, for the specific case of a uniform distribution of the temporal scale levels in units of the composed variance of the composed temporal scale-space kernels, and corresponding to the time-constants of all the primitive truncated exponential kernels being equal.
We will study three types of idealized model signals for which closed-form theoretical analysis is possible: (i) a temporal peak modelled as a set of K 0 truncated exponential kernels with equal time constants coupled in cascade, (ii) a temporal onset ramp modelled as the primitive function of the temporal peak model and (iii) a temporal sine wave. Specifically, we will analyse how the selected scale levelŝ K obtained from local extrema of temporal derivatives over scale relate to the temporal duration of a temporal peak or a temporal onset ramp alternatively how the selected scale levelsK depends on the the wavelength of a sine wave.
We will also study how good approximation the scalenormalized magnitude measure at the maximum over temporal scales is compared to the corresponding fully scaleinvariant magnitude measures that are obtained from the noncausal temporal scale concept as listed in Section 3.

Time-causal scale space based on truncated exponential kernels with equal time constants coupled in cascade
Given the requirements that the temporal smoothing operation in a temporal scale-space representation should obey (i) linearity, (ii) temporal shift invariance, (iii) temporal causality and (iv) guarantee non-creation of new local extrema or equivalently new zero-crossings with increasing temporal scale for any one-dimensional temporal signal, it can be shown (Lindeberg [57,77,78]; Lindeberg and Fagerström [81]) that the temporal scale-space kernels should be constructed as a cascade of truncated exponential kernels of the form If we additionally require the time constants of all such primitive kernels that are coupled in cascade to be equal, then this leads to a composed temporal scale-space kernel of the form corresponding to Laguerre functions (Laguerre polynomials multiplied by a truncated exponential kernel) and also equal to the probability density function of the Gamma distribution having a Laplace transform of the form Differentiating the temporal scale-space kernel with respect to time t gives see the second row in Figure 1 for graphs. The L 1 -norms of these kernels are given by The temporal scale level at level K corresponds to temporal variance τ = Kµ 2 and temporal standard deviation σ = √ τ = µ √ K.

Temporal peak
Consider an input signal defined as a time-causal temporal peak corresponding to filtering a delta function with K 0 first-order integrators with time constants µ coupled in cascade: With regard to the application area of vision, this signal can be seen as an idealized model of an object with temporal duration τ 0 = K 0 µ 2 that first appears and then disappears from the field of view, and modelled on a form to be algebraically compatible with the algebra of the temporal receptive fields. With respect to the application area of hearing, this signal can be seen as an idealized model of a beat sound over some frequency range of the spectrogram, also modelled on a form to be compatible with the algebra of the temporal receptive fields. Define the temporal scale-space representation by convolving this signal with the temporal scale-space kernel (43) corresponding to K first-order integrators having the same time constants µ where we have applied the semi-group property that follows immediately from the corresponding Laplace transforms By differentiating the temporal scale-space representation (44) with respect to time t we obtain implying that the maximum point is assumed at and the inflection points at This form of the expression for the time of the temporal maximum implies that the temporal delay of the underlying peak t max,0 = µ(K 0 − 1) and the temporal delay of the temporal scale-space kernel t max,U = µ(K − 1) are not fully additive, but instead composed according to If we define the temporal duration d of the peak as the distance between the inflection points, if furthermore follows that this temporal duration is related to the temporal duration d 0 = 2µ √ K 0 − 1 of the original peak and the temporal duration d U = 2µ √ K − 1 of the temporal scale-space kernel according to Notably these expressions are not scale invariant, but instead strongly dependent on a preferred temporal scale as defined by the time constant µ of the primitive first-order integrators that define the uniform distribution of the temporal scales.
Scale estimateK and maximum magnitude L ζζ,max from temporal peak (uniform distr) K 0K (var, γ = 3/4) L ζζ,maxmagn,postnorm γ=1 (var, γ = 3/4)K (var, γ = 1)K (Lp, p = 1)  Table 1 Numerical estimates of the value ofK at which the scale-normalized second-order temporal derivative assumes its maximum over temporal scale for a temporal peak (with the discrete expression over discrete temporal scales extended to a continuous variation) as function of K 0 and for either (i) variance-based normalization with γ = 3/4, (iii) variance-based normalization with γ = 1 and (iv) Lp-normalization with p = 1. For the case of variance-based normalization with γ = 3/4, (ii) the post-normalized magnitude measure L ζζ,maxmagn,postnorm γ=1 according to (20) and at the corresponding scale (i) is also shown. Note that the temporal scale estimatesK do for γ = 3/4 constitute a good approximation of the temporal scaleK 0 of the underlying structure and that the maximum magnitude estimates obtained at this temporal scale do for γ = 1 constitute a good approximation to a scale-invariant constant maximum magnitude measure over temporal scales.
Scale estimateK and maximum magnitude L ζ,max from temporal ramp (uniform distr)  Table 2 (columns 2-3) Numerical estimates of the value ofK at which the scale-normalized first-order temporal derivative assumes its maximum over temporal scale for a temporal onset ramp (with the discrete expression over discrete temporal scales extended to a continuous variation) as function of K 0 and for (i) variance-based normalization for γ = 1/2 and (ii) Lp-normalization for p = 2/3. (columns 4-5) Maximum magnitude values L ζ,max at the corresponding temporal scales, with the magnitude values defined by (iii) variance-based normalization for γ = 1 and (iv) Lp-normalization for p = 1. Note that for γ = 1/2 as well as for p = 2/3 the temporal scale estimatesK constitute a good approximation of the temporal scale K 0 of the underlying onset ramp as well as that the scale-normalized maximum magnitude estimates L ζ,max computed for γ = 1 and p = 1 constitute a good approximation to a scale-invariant constant magnitude measure over temporal scales.
When using temporal scale normalization by L p -normalization, the first-and second-order scale-normalized derivatives are on the other hand given by (Lindeberg [78,Equation (75) with the scale-normalization factors α n,p (µ, K) determined such that the L p -norm of the scale-normalized temporal derivative computation kernel where G 1,p and G 2,p denote the L p -norms (7) of corresponding Gaussian derivative kernels for the value of γ at which they become constant over scales by L p -normalization, and the L p -norms U t (·; µ, K) p and U tt (·; µ, K) p of the temporal scale-space kernels U t and U tt for the specific case of p = 1 are given by (41) and (42).
Temporal scale selection. Let us assume that we want to register that a new object has appeared by a scale-space extremum of the scale-normalized second-order derivative response.
To determine the temporal moment at which the temporal event occurs, we should formally determine the time where ∂ τ (L ζζ (t; µ, K)) = 0, which by our model (54) would correspond to solving a third-order algebraic equation. To simplify the problem, let us instead approximate the temporal position of the peak in the second-order derivative by the temporal position of the peak t max according to (48) in the signal and study the evolution properties over scale K of In the case of variance-based normalization for a general value of γ, we have and in the case of L p -normalization for p = 1 To determine the scaleK at which the local maximum is assumed, let us temporarily extend this definition to continuous values of K and differentiate the corresponding expressions with respect to K. Solving the equation numerically for different values of K 0 then gives the dependency on the scale estimateK as function of K 0 shown in Table 1 for variance-based normalization with either γ = 3/4 or γ = 1 and L p -normalization for p = 1.
As can be seen from the results in Table 1, when using variance-based scale normalization for γ = 3/4, the scale estimateK closely follows the scale K 0 of the temporal peak and does therefore imply a good approximate transfer of the scale selection property (14) to this temporal scalespace concept. If one would instead use variance-based normalization for γ = 1 or L p -normalization for p = 1, then that would, however, lead to substantial overestimates of the temporal duration of the peak.
Furthermore, if we additionally normalize the input signal to having unit contrast, then the corresponding timecausal correspondence to the post-normalized magnitude measure (20) is for scale estimates proportional to the temporal duration of the underlying temporal peakK ∼ K 0 very close to constant under variations of the temporal duration of the underlying temporal peak as determined by the parameter K 0 , thus implying a good approximate transfer of the scale selection property (21).

Temporal onset ramp
Consider an input signal defined as a time-causal onset ramp corresponding to the primitive function of K 0 first-order integrators with time constants µ coupled in cascade: With respect to the application area of vision, this signal can be seen as an idealized model of a new object with temporal diffuseness τ 0 = K 0 µ 2 that appears in the field of view and modelled on a form to be algebraically compatible with the algebra of the temporal receptive fields. With respect to the application area of hearing, this signal can be seen as an idealized model of the onset of a new sound in some frequency band of the spectrogram, also modelled on a form to be compatible with the algebra of the temporal receptive fields.
Define the temporal scale-space representation of the signal by convolution with the temporal scale-space kernel (43) corresponding to K first-order integrators having the same time constants µ Then, the first-order temporal derivative is given by which assumes its temporal maximum at t ramp = µ(K 0 + K − 1).
Temporal scale selection. Let us assume that we are going to detect a new appearing object from a local maximum in the first-order derivative over both time and temporal scales. When using variance-based normalization for a general value of γ, the scale-normalized response at the temporal maximum in the first-order derivative is given by When using L p -normalization for a general value of p, the corresponding scale-normalized response is where the L p -norm of the first-order scale-space derivative kernel can be expressed in terms of exponential functions, the Gamma function and hypergeometric functions, but is too complex to be written out here. Extending the definition of these expressions to continuous values of K and solving the equation numerically for different values of K 0 then gives the dependency on the scale estimateK as function of K 0 shown in Table 2 for variance-based normalization with γ = 1/2 or L p -normalization for p = 2/3. As can be seen from the numerical results, for both variancebased normalization and L p -normalization with corresponding values of γ and p, the numerical scale estimates in terms ofK closely follow the diffuseness scale of the temporal ramp as parameterized by K 0 . Thus, for both of these scale normalization models, the numerical results indicate an approximate transfer of the scale selection property (14) to this temporal scale-space model. Additionally, the maximum magnitude values according to (69) can according to Stirling's formula Γ (n + 1) ≈ (n/e) n √ 2πn be approximated by and are very stable under variations of the diffuseness scale K 0 of the ramp, and thus implying a good transfer of the scale selection property (31) to this temporal scale-space concept.

Temporal sine wave
Consider a signal defined as a sine wave This signal can be seen as a simplified model of a dense temporal texture with characteristic scale defined as the wavelength λ 0 = 2π/ω of the signal. In the application area of vision, this can be seen as an idealized model of watching some oscillating visual phenomena or watching a dense texture that moves relative the gaze direction. In the area of hearing, this could be seen as an idealized model of temporally varying frequencies around some fixed frequency in the spectrogram corresponding to vibrato.
By differentiating (74) with respect to time t, it follows that the magnitude of the nth order temporal derivative is given by Temporal scale selection. Using variance-based temporal scale normalization, the magnitude of the corresponding scalenormalized temporal derivative is given by L ζ n ,ampl = σ n L t n ,ampl = (Kµ 2 ) nγ/2 ω n 0 (1 + µ 2 ω 2 0 ) Extending this expression to continuous values of K and differentiating with respect to K implies that the maximum over scale is assumed at scalê with the following series expansion for small values of ω 0 corresponding to temporal structures of longer temporal duration Expressing the corresponding scale estimateσ in terms of dimension length and parameterized in terms of the wavelength λ 0 = 2π/ω 0 of the sine wavê σ = √τ = µ K = µ γn with the following series expansion for large λ 0 = 2π/ω 0 and γ = 1: L ζ n ,ampl,max = e −n/2 n n/2 × 1 + π 2 µ 2 n λ 2 0 + π 4 µ 4 n(3n − 10) Again we can note that the first term agrees with the corresponding scale selection property (35) for the scale-invariant non-causal Gaussian temporal scale space, whereas the higher order terms are not scale invariant. Figure 2 shows graphs of the scale estimateσ according to (82) for n = 2 and γ = 3/4 and the maximum response over scale L ζ n ,ampl,max for n = 2 and γ = 1 as function of the wavelength λ 0 of the sine wave (marked in blue). For comparison, we also show the corresponding scale estimates (34) and magnitude values (35) that would be obtained using temporal scale selection in the scale-invariant non-causal Gaussian temporal scale space (marked in brown).
As can be seen from the graphs, both the temporal scale estimateσ(λ 0 ) and the maximum magnitude L ζ,ampl,max (λ 0 ) obtained from a set of first order integrators with equal time constants coupled in cascade approach the corresponding results obtained from the non-causal Gaussian scale space for larger values of λ 0 in relation to the time constant µ of the first-order integrators. The scale estimate obtained from a set of first-order integrators with equal time constants is, however, for lower values of λ 0 generally significantly higher than the scale estimates obtained from a noncausal Gaussian temporal scale space. The scale-normalized magnitude values, which should be constant over scale for a scale-invariant temporal scale space when γ = 1 according to the scale selection property (35), are for lower values of λ 0 much higher than the scale-invariant limit value when performing scale selection in the temporal scale-space concept obtained by coupling a set of first-order integrators with equal time constants in cascade. The scale selection properties (34) and (35) are consequently not transferred to this temporal scale-space concept for the sine wave model, which demonstrates the need for using a scale-invariant temporal scale-space concept when formulating mechanisms for temporal scale selection. The scale selection properties of such a scale-invariant time-causal temporal concepts will be analysed in Section 5, and showing that it is possible to obtain temporal scale estimates for a dense sine wave that are truly proportional to the wavelength of the signal, i.e., a characteristic estimate of the temporal duration of the temporal structures in the signal.
Concerning this theoretical analysis, it should be noted that we have here for the expressions (82), (83) and (84) disregarded the rounding of the continuous valueK in (81) to the nearest integer upwards or downwards where it assumes its maximum value over temporal scales. Thereby, the graphs in Figure 2 may appear somewhat different if such quantization effects because of discrete temporal scale levels are also included. The lack of true temporal scale invariance will, however, still prevail.
Concerning the motivation to the theoretical analysis in this section, while the purpose of this analysis has been to investigate how the temporal scale estimates depend on the frequency or the wavelength of the signal, it should be emphasized that the primary purpose has not been to develop a method for only estimating the frequency or the wavelength of a sine wave. Instead, the primary purpose has been to carry out a closed-form theoretical analysis of the properties of temporal scale selection when applied to a model signal for which such closed-form theoretical analysis can be carried out. Compared to using e.g. a Fourier transform for estimating the local frequency content in a signal, it should be noted that the computation of a Fourier transform requires a complementary parameter -a window scale over which the Fourier transform is to be computed. The frequency es-timate will then be an average of the frequency content over the entire interval as defined by the window scale parameter. Using the the proposed temporal scale selection methodology it is on the other hand possible to estimate the temporal scale without using any complementary window scale parameter. Additionally, the temporal scale estimate will be instantaneous and not an average over multiple cycles of e.g. a periodic signal, see also the later experimental results that will be presented in Section 7 in particular Figure 7. In this section, we will analyse the scale selection properties for the time-causal scale-space concept based on convolution with the scale-invariant time-causal limit kernel.
The analysis starts with a detailed study of a sine wave, for which closed-form theoretical analysis is possible and showing that the selected temporal scale levelσ measured in units of dimension time [time] according toσ = √τ will be proportional to the wavelength of the signal, in accordance with true scale invariance. We also show that despite the discrete nature of the temporal scale levels in this temporal scale-space concept, local extrema over scale will nevertheless be preserved under scaling transformations of the form λ 1 = c j λ 0 , with c denoting the distribution parameter of the time-causal limit kernel.
Then, we present a general result about temporal scale invariance that holds for temporal derivatives of any order and for any input signal, showing that under a temporal scaling transformation of the form t = c j t, local extrema over scales are preserved under such temporal scaling transformations with the temporal scale estimates transforming according to τ = c j τ . We also show that if the scale normalization power γ = 1 corresponding to p = 1, the scalenormalized magnitude responses will be preserved in accordance with true temporal scale invariance.

Time-causal temporal scale space based on the scale-invariant time-causal limit kernel
Given the temporal scale-space model based on truncated exponential kernels (36) having a composed Fourier transform of the form and as arises from the assumptions of (i) linearity, (ii) temporal shift invariance, (iii) temporal causality and (iv) noncreation of new local extrema or equivalently zero-crossings with increasing scale, it is more natural to distribute the temporal scale levels logarithmically over temporal scales so that the distribution in terms of effective temporal scale τ ef f = log τ (Lindeberg [59]) becomes uniform. This implies that time constants of the individual first-order integrators should for some c > 1 be given by (Lindeberg [77,78]) Specifically, if one lets the number of temporal scale levels tend to infinity with the density of temporal scale levels becoming infinitely dense towards τ → 0, it can be shown that this leads to a scale-invariant time-causal limit kernel having a Fourier transform of the form (Lindeberg [78, Sec-

Temporal sine wave
Consider an input signal defined as a sine wave and taken as an idealized model of a oscillating signal with temporal structures having characteristic temporal duration λ 0 = 2π/ω 0 . For the time-causal temporal scale-space defined by convolution with the time-causal semi-group Ψ (t; τ, c), the temporal scale-space representation L(t; τ, c) is given by where the magnitude |Ψ (ω; τ, c)| and the argument argΨ (ω; τ, c) of the Fourier transformΨ (ω; τ, c) of the time-causal limit kernel are given by Thus, the magnitude on the nth order temporal derivative is given by L t n ,ampl = ω n |Ψ (ω; τ, c)| (95) and for the nth order scale-normalized derivate based on variance-based scale normalization the amplitude as function of scale is Figure 3 shows graphs of the variation of this entity as function of temporal scale for different angular frequencies ω 0 , orders of temporal differentiation n and the distribution parameter c. As can be seen from the graphs, the maxima over scales are assumed at coarser scales with increasing wavelength of the sine wave. The maxima over temporal scale are also assumed at coarser scales for second-order derivatives than for first-order derivatives. Specifically, when γ = 1 the magnitude values at the local extrema over scale are constant over scale, which implies a transfer of the scale selection property (10) to this temporal scale-space model. Notably, this situation is in clear contrast to the situation for the temporal scale-space generated by first-order integrators with equal time constants coupled in cascade. In Figure 2 it was shown that because of the lack of true temporal scale invariance of that temoral scale-space concept, the maximum magnitude values are not constant over scales for γ = 1 as they should be according to the scale-invariant scale selection property (10).
When choosing lower values of γ as motivated from the determination of the parameter γ for scale selection in a Gaussian scale space to make the scale estimate reflect the width of a Gaussian peak for second-order derivatives or reflect the width of a diffuse ramp for first-order derivatives, which leads to γ = 1/2 for first-order derivatives (27) and γ = 3/4 for second-order derivatives (16), the local extrema over scale are moved to finer scales (see Figure 4). Then, however, the maximum magnitude values are no longer the same for sine waves of different frequencies, implying that a complementary magnitude normalization step is necessary (see Section 5.3 for additional details). Corresponding results as in Figure 3 above but with (left column) γ = 1/2 for first-order derivatives and (right column) γ = 3/4 for second-order derivatives. Note that the use of scale normalization powers γ < 1 implies that the maxima over temporal scales are moved to finer temporal scales and that the (uncompensated) maximum magnitude responses are no longer scale invariant. If we treat τ as a continuous variable and differentiate with respect to τ we obtain Figure 5 show graphs of how the scale estimateσ obtained by setting the derivative with respect to temporal scale to zero increases linearly with the wavelength λ 0 of the signal, with different slopes of the linear curve depending on the order of temporal differentiation and the value of the scale normalization parameter γ. This overall linear scaling behaviour can directly be proved by rewriting the expression ∂ τ (log L ζ n ,ampl ) = 0 in (98) into Since this expression is a direct function of the dimensionless entity τ ω 2 0 , it follows that the temporal scale estimates will be of the form for some function ϕ(nγ, c), and thus obeying temporal scale invariance in the sense that the scale estimate in dimension length is proportional to the wavelength of the signal Notice how this situation is in contrast to the results of scale selection in the temporal scale-space concept obtained by coupling first-order integrators with equal time constants in cascade, where the scale estimate for a sine wave is not directly proportional to the wavelength of the temporal signal, but also affected by a wavelength dependent temporal scale bias (see Equation (82) and Figure 2 in Section 4.4). Table 3 shows numerical values of the differences between the results for different orders of differentiation n and different values of γ. These numerical entities become particularly illuminating by forming the ratioσ/ √ nγ as shown in Table 4. For a non-causal Gaussian scale space, this ratio should be equal to one for all combinations of n and γ (see Equation (33)). For this non-causal temporal scale space, we can, however, note that the deviation from one increases both with larger values of the distribution parameter 2 c and with increasing order of temporal differentiation n, which both lead to larger degrees of temporal asymmetry due to the non-causal temporal dimension.
In the essential proportionality of the scale estimateσ to the wavelength λ 0 of the signal according to (101), the main component of the scale selection property (34) is thereby transferred to this temporal scale-space concept, although the proportionality constant has to be modified depending on the value of the temporal scale distribution parameter c, the order of temporal differentiation n and the scale normalization parameter γ. Scale estimatesσ for c = 2 n = 1 n = 2 γ = 1 1.43 3.17 γ = γn 0.83 2.17 Table 3 Scale estimatesσ computed from local extrema over scale for a sine wave with angular frequency ω 0 = 1 and different orders of temporal differentiation n and different scale normalization parameters γ with γn = 1/2 for n = 1 and γn = 3/4 for n = 2, and different values of the distribution parameter c. (For c = √ 2 the time-causal limit kernel has been approximated by the slowest K = 32 temporal smoothing stages and for c = 2 by the slowest K = 12 temporal smoothing stages.)

Ratiosσ/
√ nγ for c = √ 2 n = 1 n = 2 γ = 1  Table 4 Ratios between the scale estimatesσ in Table 3 and √ nγ computed from local extrema over scale for a sine wave with angular frequency ω 0 = 1. For the corresponding entities obtained from scale selection for a sine wave in a non-causal Gaussian scale space, this ratio is equal to one for all combinations of n and γ. These scale selection entities thus reveal a larger deviation from a Gaussian behaviour both for larger values of c and for temporal derivatives of higher order. over temporal scales for some pair (ω 0 , τ 0 ) and that the derivative with respect to temporal scale is thereby zero Let us next assume that we feed in a different sine wave with wavelength λ 1 = c j λ 0 for some integer j (for the same value of c as used in the definition of the time-causal limit kernel) and corresponding to ω 1 = c −j ω 0 with its matching scale τ 1 = c 2j τ 0 . Then, it holds that This result implies that the sign of the derivate with respect to temporal scale is preserved between matching angular frequencies and scales (ω 0 , τ 0 ) and (ω 1 , τ 1 ). Specifically, local extrema over temporal scales are preserved under uniform scaling transformations of the temporal domain t = c j t, implying scale covariance of the temporal scales that are selected from local extrema over scales of scale-normalized temporal derivatives. Note also that although the analysis in Equation (103) is performed based on a temporary extension of τ into a continuous variable, the scale covariance still holds when the continuous function is sampled into a discrete set of temporal scale levels. 3

General scale invariance property under temporal scaling transformations
In Lindeberg [78,Appendix 3] it is shown that for two temporal signals f and f that are related by a temporal scaling transform f (t ) = f (t) for t = c j −j t with the corresponding transformation between corresponding temporal scale levels τ = c 2(j −j) τ , the scale-normalized temporal derivatives defined by either L p -normalization or variance-based normalization in the scale-space representation defined by convolution with the time-causal limit kernel Ψ (t; τ, c) are for any temporal input signal f related according to L ζ n (t ; τ , c) = c (j −j)n(γ−1) L ζ n (t; τ, c) = c (j −j)(1−1/p) L ζ n (t; τ, c).
This result specifically implies that the scale-space signatures, which are the graphs that show the variation in the strength of scale-normalized derivatives over scale, will be rescaled copies of each other for signals that are related by a uniform scaling transformation of the temporal domain. Specifically, local temporal scale estimatesτ andτ as determined from local extrema over temporal scales in the two temporal domains will be assumed at corresponding 3 A formal proof of the transfer of this preservation property of local extrema over temporal scales from a temporary extension of the temporal scale parameter τ into a continuous variable back into a restricted discrete set of temporal scale levels can be stated as follows: From Equation (103) it follows that the continuous temporal scale-space signatures for the two sine waves of wavelengths λ 0 and λ 1 = c j λ 0 will increase and decrease respectively at corresponding matching temporal scale levels τ 0 and τ 1 = c 2j τ 0 . If we next sample these scale-space signatures at some discrete set of temporal scale levels τ 0,k = c 2k and τ 1,k = c 2j c 2k , then it follows that the discrete maxima over temporal scales will also be related according to τ 1,max = c 2j τ 0,max . This preservation property of local extrema does, however, only hold for temporal scaling factors S that are integer powers of the distribution parameter c, i.e., only S = c j . Alternatively, this preservation property can also be derived from the more general scale invariance property under temporal scaling transformations (104) that is stated in next section. temporal scale levels and will thus be transformed in a scalecovariant way for any temporal scaling transformation of the form t = c j −j t. In units of the temporal variance, it holds that τ = c 2(j −j)τ (105) and in units of the temporal standard deviation If γ = 1 corresponding to p = 1, the magnitude values at corresponding temporal scale levels will be equal. If γ = 1 corresponding to p = 1, the magnitude values will be related according to (104). Thereby, this expression provides a way to normalize maximum strength measures between local extrema over scales assumed at different temporal scales as obtained e.g. in the scale-space signatures shown in Figure 4. Note that by this construction we have been able to transfer the temporal scale invariance property (9) and (10) that holds for a non-causal Gaussian temporal scale-space concept to also hold for a time-causal temporal scale-space concept, which is a novel type of theoretical construction. This property is, however, restricted to the temporal scale-space concept based on convolution with the time-causal limit kernel and does, for example, not hold for the time-causal temporal scale-space concept based on convolution with a cascade of truncated exponential kernels having equal time constants and corresponding to a uniform distribution of the temporal scale levels in units of the composed temporal variance.

Influence of discrete temporal scale levels on the theoretical analysis
In the theoretical analysis of scale selection properties of (i) a temporal peak in Section 4.2, (ii) a temporal onset ramp in Section 4.3 and (ii) a temporal sine wave in Section 4.4 and Section 5.2, we did first compute closed-form expressions for how the scale-normalized temporal magnitude measures depend upon the temporal scale levels τ K = Kµ 2 according to (62), (69) and (79) for the time-causal scalespace concept based on truncated exponential kernels with equal time constant coupled in cascade or how the scalenormalized magnitude measure depends on the temporal scale level τ K = c 2K τ 0 according to (96) for the time-causal temporal scale-space concept based on the scale-invariant limit kernel: M sine,uni (K) = L ζ n ,ampl (K) Then, to compute the temporal scale levels at which the scale-normalized derivative responses assumed their maximum values over temporal scales, we temporarily extended these magnitude measures from being defined over discrete integer temporal scale levels K to a continuum over K, to be able to differentiate the closed-form expressions with respect to the temporal scale level. A general question that could be raised in this context therefore concerns how good approximation the results from the continuous approximation of local extrema over scales are with respect to a setting where the temporal scale levels are required to be discrete. A common property of the four types of scale-space signatures according to equations (107)- (109) and shown in Figure 6 is that they are unimodal, i.e., they assume a single maximum over temporal scales and do first increase and then decrease. Thereby, when the continuous variable K c with its associated maximum over temporal scalesK c obtained from a continuous analysis is in a second stage restricted to be discrete, it follows that the discrete maximum over discrete temporal scalesK d is guaranteed to be assumed at either the nearest lower or the nearest higher integer. Thus, we obtain the discrete temporal scale estimate by rounding the continuous scale estimateK c to either the nearest lower or the nearest higher integer.
Whether the value should be rounded upwards or downwards depends on how close the continuous estimateK c is to the nearest downwards vs. upwards integers and on the local degree of asymmetry of the scale-space signature around the maximum over temporal scales.
When implementing and executing a temporal scale selection algorithm in practice, the situation can on the other hand be reverse. Given a set of discrete temporal scale levels, we may detect a local maximum over temporal scales at Temporal peak for K 0 = 5 Scale-space signature M peak,uni (K)  (43) with temporal duration determined by K 0 = 5 for which the second-order scale-normalized temporal derivative response at the peak varies according to (107) here with γ = 3/4, (second row) Time-causal onset ramp model according to (66) with temporal duration determined by K 0 = 5 for which the first-order scale-normalized temporal derivative response at the ramp varies according to (108) here with γ = 1/2, (third row) Temporal sine wave according to (73) with angular frequency ω 0 = 1 for which the amplitude of the second-order temporal derivative response varies according to (109) here with γ = 1 and (bottom row) Temporal sine wave according to (91) for which the amplitude of the second-order temporal response measure varies according to (110) here with γ = 1 . The scale-space signatures in the first three rows have been computed using the time-causal temporal scale-space concept based on truncated exponential kernels with equal time constants µ = 1 coupled in cascade, whereas the scale-space signature in the bottom row has been computed computed using the time-causal temporal scale-space concept obtained by convolution with the scale-invariant limit kernel with distribution parameter c = 2. Note specifically that the shape of the scale-space signature in the third row is different from the shape in the fourth row because of the uniform distribution of temporal scale levels K in the third row and the logarithmic distribution in the fourth row. (Horizontal axis for the figures in the left column: time t.) (Horizontal axis for the figures in the right column: temporal scale level K.) some discrete temporal scale levelK d . If we would like to use this temporal scale estimate for estimating the temporal duration of the underlying temporal structure that gave rise to the response, e.g. according to the methodology outlined in appendix C, we may on the other hand would like to compute a better continuous estimateK c of the temporal scale level than as restricted by the discrete temporal scale levels.
A straightforward way of computing a more accurate temporal scale estimate in such a situation is by interpolating a parabola over the measurements over the temporal scale levels in an analogous way as subresolution spatial scale estimates can be obtained over a spatial scale-space representation [66,80,89]. Let (x 0 , y 0 ) denote the scale level and the magnitude measure at the discrete maximum and let (x 0 −1, y −1 ) and (x 0 +1, y 1 ) denote the corresponding scale level and magnitude measure at the nearest lower and upper temporal scales, respectively. Assuming the following form of the interpolating function the interpolation coefficients become with the corresponding subresolution estimate of the maximum over scaleŝ . (115) Note that the correction offset ∆x = −b/a is restricted to the interval ∆x ∈ [−1/2, 1/2] implying that the location x 0 of the discrete maximum is guaranteed to be on the sampling grid point x i nearest to the subresolution estimatex.

Temporal scale selection for 1-D temporal signals
To illustrate the derived scale selection properties, we will in this section show the result of applying temporal scale selection to different types of purely temporal signals. The bottom rows in Figure 7 shows a one-dimensional model signal having a temporally varying frequency of the form defined such that the local wavelength increases with time t. Figure 8 shows the temporal scale-space representation of the scale-normalized second-order temporal −L ζζ as function of scale for the maximally scale-invariant choice of γ = 1 corresponding to p = 1. Note how structures in the signal of longer temporal duration give rise responses at coarser temporal scales in agreement with the derived theoretical properties of the scale-covariant scale-space concepts over a time-causal vs. a non-causal temporal domain.
In the top rows in Figure 7 we show the results of applying local temporal scale selection by detecting local maxima over both time and scale of the scale-normalized secondorder temporal derivative −L ζζ for the scale-calibrated choice of γ = 3/4 corresponding to p = 2/3. Each detected scalespace extremum has been marked by a star at the point in scale-space (t,σ) = (t,

√τ
) at which the local maximum over temporal scale was assumed. Such temporal scale selection has been performed using two types of temporal scalespace concepts: (i) based on the time-causal scale-space representation corresponding to convolution with the scale-invariant limit kernel with distribution parameter c = √ 2 approximated by a finite number of the at least K = 8 slowest primitive smoothing steps at the finest level of scale or (ii) based on the non-causal Gaussian kernel using 5 temporal scale levels per scale octave.
The discrete implementation of the time-causal temporal scale-space representation corresponding to convolution with the scale-invariant limit kernel has been based on recursive filters over time according to the methodology described in Lindeberg [78, section 6.2] whereas the discrete implementation of the non-causal Gaussian temporal scalespace concept has been based on the discrete analogue of the Gaussian kernel described in Lindeberg [57,58]. Discrete implementation of scale-normalized temporal derivatives has in turn been based on discrete l p -normalization according to the methodology outlined in [78, section 7].
As can be seen from the results in the top rows in Figure 7, the temporal scale estimatesσ increase proportional to the local wavelength λ according to the derived scaleinvariant properties of the temporal scale-space concepts based on convolution with the time-causal limit kernel or the noncausal Gaussian kernel. The scale estimates obtained using the time-causal vs. the non-causal temporal scale space concepts are, however, not equal in units of the standard deviationσ of the temporal scale-space kernel -see also the theoretical analysis in section 5 with specifically the numerical comparisons in Table 4. Thus, different scale calibration factors are needed to transform the temporal scale estimates in units of the temporal standard deviation to units of the temporal duration of the signal for the time-causal vs. the non-causal Gaussian temporal scale-space concepts -compare with the theoretical analysis in appendix C.
Another qualitative difference that can be noted with regard to temporal scale selection in a time-causal temporal scale-space representation vs. a non-causal Gaussian temporal scale-space representation is that any measurement performed in a time-causal temporal scale-space concept is associated with an inherent temporal delay δ, whereas the tem-Scale-space extrema from the time-causal limit kernel Scale-space extrema from the non-causal Gaussian kernel    Fig. 10 The result of performing temporal delay compensation of the scale-space extrema shown in Figure 9 by correcting the temporal momentt of the scale-space extremum by an estimate of the temporal delay computed from the position of the temporal maximum of the temporal scale-space kernel according to [78,Section 4]. Note that although the computations by necessity have to be associated with a temporal delay, we can nevertheless compute a good estimate of when the underlying event occurred that gave rise to the feature response.
poral delay can be defined to be zero for a non-causal Gaussian temporal scale-space. Thus, the local extrema over scales will be assumed later with increasing temporal scale levels as induced by temporal structures of having longer temporal duration. Varying the parameter q in the scale calibration criteria (14) and (25) to values of q < 1 provides a straightforward way of enforcing responses to be obtained at finer temporal scales and thereby implying shorter temporal delays, at the potential cost of a larger likelihood of false positive re- Time-causal scale-space −L ζζ Non-causal scale-space −L ζζ Fig. 11 Illustration of the temporal dynamics over multiple temporal scales that arises when a temporal scale-space representation responds to a temporal peak with temporal standard deviation σ 0 = 32 for (left column) the time-causal representation corresponding to convolution with the time-causal limit kernel and (right column) the non-causal Gaussian temporal scale space. (top row) Graphs of the input kernels for the cases of a time-causal peak and a non-causal peak, respectively. (bottom row) Scale-space maps of the scale-normalized second-order temporal derivative −L ζζ . Note that if we slice the time-causal scalespace map vertically at a temporal moment before the full development of the temporal scale-space maximum at the temporal scale corresponding to the temporal scale of the peak, we will get earlier temporal responses at finer temporal scales, whereas if we slice the time-causal scale-space map at a temporal moment after the full development of the temporal scale-space maximum, we will get later temporal responses at coarser temporal scales. When handling multiple temporal scale levels in a time-causal real-time situation, it it therefore natural to include explicit mechanisms for tracking and handling how the temporal structures evolve over temporal scales. (Horizontal axis: time t.) (Vertical axis in bottom row: effective temporal scale log τ .) sponses by not detecting the underlying temporal structures at the same temporal scales as they occur. Yet another side effect of the longer temporal delays at coarser temporal scales is that multiple local responses over scales may be obtained with respect to the same underlying temporal structure, with first responses obtained at finer temporal scales followed by later responses at coarser temporal scales. Because of the temporal shift caused by different temporal delays between adjacent temporal scale levels, scale-space extrema detection over a local 3 × 3 neighbourhood may not detect a single extremum over temporal scales as for the non-causal Gaussian temporal scale-space concept. Therefore, explicit handling of different temporal delays at different temporal scale levels is needed when performing temporal scale selection in a time-causal temporal scale-space representation. The presented theory of temporal scale selection properties is intended to be generally ap-plicable with respect to different such strategies for handling the temporal delays in specific algorithms.

Post-filtering of responses at adjacent temporal scales
In view of the behaviour of image structures over temporal scales illustrated in Figure 8, one way of suppressing multiple responses to the same underlying structure at different temporal scales is by performing an additional search around each scale-space extremum as follows: If a point (t,τ ) at temporal scale level with temporal scale index k is a scale-space maximum, perform an additional search at the nearest finer temporal scale level k − 1 to previous temporal moments t − j as long as the scale-normalized values monotonically increase. When the monotone increase stops and a local temporal maximum has been found, then compare if the temporal maximum value at the nearest finer temporal scale is greater than the temporal maximum value at the current scale. If so, suppress the scale-space maximum at the current scale. In a corresponding manner, perform a search at the next coarser temporal scale k + 1 to the following temporal moments t + j as long as the scale-normalized values monotonically increase. When the monotone increase stops and a local temporal maximum has been found, then compare if the temporal maximum value at the next coarser temporal scale is greater than the temporal maximum value at the current scale. If so, suppress the scale-space maximum at the current scale. By this type of straightforward scale-space tracking over adjacent temporal scales, a single response will be obtained to the same underlying structure as illustrated in Figure 9. Additionally, a more accurate temporal scale estimate can be computed by performing the parabolic interpolation according to (111) and (115) over the nearest backward and forward temporal maxima at the adjacent finer and coarser temporal scales as opposed to an interpolation over temporal scales at the same temporal moment as at which the temporal scale-space maximum was assumed.
Note that in a real-time situation, the necessary information needed to perform a search to the past can be stored by a process that records the value of temporal maxima at each temporal scale. In a corresponding manner, later deletion or subresolution interpolation of the scale estimate of a scale-space maximum can only be performed when time has passed to the location of the next temporal maximum at the nearest coarser temporal scale. If a preliminary feature response has been obtained at any temporal scale while a potential response at the nearest coarser temporal scale is not available yet because of its longer temporal scale, a realtime system operating over multiple scales should preferably be designed with the ability to correct or adjust preliminary measurements when more information at coarser scales becomes available -see also Figure 11 for an illustration of the underlying temporal dynamics that arises when processing signals at multiple scales using a time-causal temporal scale-space representation.
While these descriptions have been given regarding scalespace maxima and local maxima, the procedure for handling scale-space minima and local minima is analogous with the polarity of the signal reversed.

Temporal delay compensation
In Figure 10 we have additionally adjusted the temporal location of every temporal scale-space maximum by an estimate of the temporal delay δ computed from the location of the temporal maximum of the underlying temporal scalespace kernel according to [78,Section 4]. Note that although any temporal event detected at a coarser temporal scale will by necessity be associated with non-zero temporal delay, we can nevertheless retrospectively compute a good estimate of when the underlying event occurred that gave rise to the registered feature response.

Temporal scale selection for a real measurement signal
While a main purpose of this article is to develop a theory of temporal scale selection to be used in conjunction with a spatio-temporal scale-space concept for video analysis or a spectro-temporal scale-space concept for audio analysis, we argue that this theory is applicable to much larger classes of time-dependent measurement signals. For the purpose of isolating the effect to a purely one-dimensional measurement signal, we do in Figure 12 show the result of applying corresponding temporal scale selection to a real measurement signal showing hourly measurements of the temperature at a weather station. Note how the temporal scale selection method based on scale-space extrema is able to extract the coarse scale temperature peaks although the signal contains substantial high-amplitude variations at finer scales.
For the purpose of having the local feature responses being less dependent on the local phase of the signal than for either first-or second-order temporal derivatives L ζ or L ζζ , we do in Figure 13 show the result of computing a quasi quadrature measure derived in [79] to for constitute an improved version of an earlier proposed quasi quadrature measure in Lindeberg [64].  Table 5 Experimental results of the accuracy of the temporal scale estimates when detecting temporal scale-space extrema of the scalenormalized derivative −L ζζ in a temporal peak defined as a timecausal limit kernel with temporal standard deviation σ 0 = √ τ 0 for distribution parameter c and then detecting the strongest temporal scalespace maximum with temporal scale estimateσ = √τ also using the same value of the distribution parameter c and for two ways of defining temporal scale-normalized derivatives by either (i) discrete lpnormalization or (ii) variance-based normalization.  Table 6 Experimental results of the accuracy of the temporal scale estimates when detecting temporal scale-space extrema of the scalenormalized derivative L ζ in a temporal onset ramp defined as the primitive function of a time-causal limit kernel with temporal standard deviation σ 0 = √ τ 0 for distribution parameter c and then detecting the strongest temporal scale-space maximum with temporal scale estimatê σ = √τ also using the same value of the distribution parameter c and for two ways of defining temporal scale-normalized derivatives by either (i) discrete lp-normalization or (ii) variance-based normalization.

Temporal scale estimates from temporal onset ramps
Temporal scale estimates from non-causal Gaussian scale space temporal peak temporal onset ramp  Table 7 Experimental results of the accuracy of the temporal scale estimates for the non-causal Gaussian temporal scale-space concept when applied to the detection temporal scale-space maxima of the scale-normalized derivative −L ζζ for a temporal peak and to the scalenormalized derivative L ζ for onset ramp detection for two ways of defining temporal scale-normalized derivatives by either (i) discrete lpnormalization or (ii) variance-based normalization. Note how this operator responds to different types of temporal structures at different temporal scales with a particularly strong response due to the temporal variations caused by the daily and annual temperature cycles. This effect becomes more immediately noticeable if we complement the above temporal scale selection method with a complementary assumption about stationarity of the signal and sum up the scale-normalized feature responses of the quasi quadrature measure over time for every temporal scale level. In the graphs shown in Figure 14 we do then obtain two major peaks over temporal scales, with the finer scale peak corresponding to the daily temperature cycle and the coarser scale peak to the annual temperature cycle.
Note that the purpose of this experiment is not primarily to develop an algorithm for detecting periodic variations in a signal but to illustrate that the temporal scale selection mechanism produces intuitively reasonable temporal scale estimates for a real-world 1-D signal with known properties.

Numerical accuracy of the temporal scale estimates
To investigate how well the temporal scale estimates generated by the resulting temporal scale selection mechanism reflect the temporal scale in the underlying temporal signal, we generated temporal model signals defined as either (i) a temporal peak modelled as time-causal limit kernel with temporal scale τ 0 and distribution parameter c for different values of τ 0 and c and (ii) a temporal onset ramp modelled as the primitive function of the temporal peak model. Then, we detected scale-space extrema of the second-order scalenormalized temporal derivative −L ζζ for peak detection or scale-space extrema of the first-order scale-normalized temporal derivative L ζ for onset detection, using the time-causal temporal scale-space concept corresponding to convolution with the time-causal limit kernel for the same value of c and resulting in temporal scale estimatesσ = √τ . Table 5 and Table 6 show results from this experiment for the time-causal temporal peak and the temporal onset ramp models, respectively. Note how well the temporal scale estimates derived for truncated exponential kernels with a uniform distribution of the temporal scale levels in section 4.2 and section 4.3 do also generalize to truncated exponential kernels with a logarithmic distribution of the temporal scale levels. Table 7 shows corresponding results for the non-causal Gaussian scale-space concept applied to non-causal temporal peak and temporal onset ramp models, respectively, and again with a very good agreement between the local scale estimatesσ in relation to the inherent temporal scale σ 0 in the signal.
For this experiment, we can specifically note that variancebased temporal scale normalization does on average lead to slightly more accurate temporal scale estimates compared to discrete l p -normalization, which can be contrasted to previous results regarding spatial scale selection in hybrid pyramids by Lindeberg and Bretzner [80], where l p -normalization lead to much more accurate scale selection results compared to variance-based normalization.

Temporal scale selection in spatio-temporal video data
In this section, we will develop a basic proof of concept of applying the proposed theory and methodology for selecting local temporal scales in video data based on a small set of specific spatio-temporal feature detectors formulated in terms of spatio-temporal differential invariants. A more detailed treatment of this topic with examples for more general families of differential expressions for spatio-temporal scale selection with associated spatio-temporal image models for scale calibration is presented in a companion paper [79].

Spatio-temporal receptive field model
For applying the proposed framework for temporal scale selection to spatio-temporal video data, we follow the approach with idealized models of spatio-temporal receptive fields of the form as previously derived, proposed and studied in Lindeberg [70,72,78] where x = (x 1 , x 2 ) T denotes the image coordinates, t denotes time, s denotes the spatial scale, τ denotes the temporal scale, v = (v 1 , v 2 ) T denotes a local image velocity, -Σ denotes a spatial covariance matrix determining the spatial shape of an affine Gaussian kernel g(x; s, Σ) = denotes a spatial affine Gaussian kernel that moves with image velocity v = (v 1 , v 2 ) in space-time and h(t; τ ) is a temporal smoothing kernel over time.
and we specifically here choose as temporal smoothing kernel over time either (i) the time-causal temporal scale-space kernel corresponding to a set of first-order integrators with equal time constants coupled in cascade (37) with τ = Kµ 2 or (ii) the time-causal limit kernel defined via its Fourier transform of the form (90). For simplicity, we shall in this treatment restrict ourselves to space-time separable receptive fields obtained by setting the image velocity to zero v = (v 1 , v 2 ) = (0, 0) and to receptive fields that are rotationally symmetric over the spatial domain as obtained by setting the spatial covariance matrix to a unit matrix Σ = I. The resulting spatiotemporal receptive fields that we then obtain correspond to complementing our time-causal temporal scale-space concepts studied in Sections 4-5 with a rotationally symmetric spatial Gaussian scale-space concept over the spatial domain.
Specifically, the natural way of expressing spatio-temporal scale selection mechanisms within this space-time separable spatio-temporal scale-space concept L(x 1 , x 2 , t; s, τ ) = (T (·, ·, ·; s, τ ) * f (·, ·, ·)) (x 1 , x 2 , t; s, τ ) is by studying scale-normalized partial derivates of the form (Lindeberg [78,Section 8.5,Equation (108)] where the factor s (m1+m2)γs/2 transforms the regular partial spatial derivatives to corresponding scale-normalized spatial derivatives with γ s denoting the spatial scale normalization parameter (Lindeberg [66]) and the factor α n (τ ) is the scale and G n,γτ denotes the L p -norm of the non-causal temporal Gaussian derivative kernel for the γ τ -value for which this L P -norm becomes constant over temporal scales.

Differential entities for spatio-temporal scale selection
Inspired by the way neurons in the lateral geniculate nucleus (LGN) respond to visual input (DeAngelis et al [16,15]), which for many LGN cells can be modelled by idealized operations of the form (Lindeberg [72,Equation (108) let us consider the following differential entities (Lindeberg [78,Section 8.4, Equations (95)-(96)]) which correspond to first-and second-order temporal derivatives of the spatial Laplacian operator and study the corresponding scale-normalized spatio-temporal derivative expressions for γ s = 1: Notably, we do not focus on extending the previously established use of the spatial Laplacian operator for spatial scale selection to a spatio-temporal Laplacian operator for spatiotemporal scale selection, since the most straightforward way of defining such an operator ∇ 2 (x,y,t) L = L xx +L yy +κ 2 L tt for some κ is not covariant under independent rescaling of the spatial and temporal coordinates as occurs if observing the same scene with cameras having independently different spatial and temporal sampling rates. The differential entities ∂ t,norm (∇ 2 (x,y),norm L) and ∂ tt,norm (∇ 2 (x,y),norm L) are on the other hand truly covariant under independent rescalings of the spatial and temporal dimensions and therefore better candidates to be used as primitives in spatio-temporal scale selection algorithms.

Response to a localized Gaussian blink
Consider a local idealized spatio-temporal image pattern defined as the combination of a rotationally symmetric Gaussian blob g(x, y; s 0 ) over the spatial domain and a timecausal temporal peak U (t; µ, K 0 ) of the form (43) over the temporal domain If we define the spatio-temporal scale-space representation of this spatio-temporal image pattern of the form (122) with the temporal scale-space kernel chosen as the composed kernel U (t; µ, K) corresponding to a cascade of first-order integrators with equal time constants coupled in cascade (120), then it follows from the semi-group property of the spatial Gaussian kernel and the semi-group property (45) of the time-causal temporal kernel U (t; µ, K) that the spatiotemporal scale-space representation will be of the form L(x, y, t; s, τ ) = g(x, y; s 0 + s) U (t; µ, K 0 + K). (132) The second-order temporal derivative of the spatial Laplacian of the Gaussian. Specifically, the scale-normalized differential entity ∂ tt,norm (∇ 2 (x,y),norm L) constituting an idealized model of a "lagged" LGN cell [78, Figure 3(right)] will by a combination of the Laplacian response of Gaussian ∇ 2 g(x, y; s) = (x 2 + y 2 − 2s)/s 2 g(x, y; s), the second-order temporal derivative of L tt (t; µ, K) of a time-causal peak in Equation (47) and the temporal scale normalization operation in Equation (54) assume the form ∂ tt,norm (∇ 2 (x,y),norm L) (x 2 + y 2 − 2(s 0 + s)) (s 0 + s) 2 g(x, y; s 0 + s) U (t; µ, K 0 + K).
Specifically, based on previously established scale selection properties of the spatial Laplacian of the Gaussian (Lindeberg [66,74]) and the second-order scale-normalized temporal derivatives of the time-causal scale space (Section 4.2), it follows that this spatio-temporal differential entity will for spatial and temporal scale normalization powers γ s = 1 and γ τ = 3/4, respectively, assume its local extremum over both spatial and temporal scales at spatial scalê and at a temporal scale that is a good approximation of the temporal scale of the temporal peakK ≈ K 0 corresponding to (see Table 1) Thus, simultaneous spatio-temporal scale selection using the differential entity ∂ tt,norm (∇ 2 (x,y),norm L) applied to the model signal (131) will estimate both the spatial extent and the temporal duration of the Gaussian blink.
The determinant of the spatio-temporal Hessian. For general values of the spatial and temporal scale normalization parameters γ s and γ τ , the determinant of the spatio-temporal Hessian is given by det H (x,y,t),norm L = s 2γs τ γτ (L xx L yy L tt + 2L xy L xt L yt In the specific case when the spatio-temporal scale-space representation of a time-causal Gaussian blink is of the form (132), if we restrict the analysis to the spatial origin (x, y) = (0, 0) where g x = g y = 0 and to the temporal maximum point t max where h t = 0, implying that L xy = L xt = L yt = 0, it follows that the determinant of spatio-temporal Hessian at the spatio-temporal maximum reduces to the form det H (x,y,t),norm L (x,y)=(0,0),t=tmax = s 2γs g g xx g yy (x,y)=(0,0) ( we can observe that the spatial and temporal scale selection properties of the determinant of the spatio-temporal Hessian will be different from the scale selection properties of the second-order temporal derivative of the spatial Laplacian. In a companion paper (Lindeberg [79]), it is shown that for a corresponding non-causal Gaussian temporal scale-space concept, spatial and temporal scale normalization parameters equal to γ s = 5/4 and γ τ = 5/4 lead to scale estimateŝ s andτ corresponding to the spatial and temporal extentŝ s = s 0 andτ = τ 0 of a Gaussian blink.
By performing a corresponding study of the temporal scale selection properties of the purely temporal component of this expression for the specific case of time-causal temporal scale-space representation based on a uniform distribution of the intermediate temporal scale levels as previously done for the temporal scale selection properties of the second-order temporal derivative of a temporal peak in Table 1, we obtain the results shown in Table 8. The column labelledK shows that the maximum over temporal scales is obtained at a temporal scale level near the temporal scale of the underlying temporal peak whereas the column labelled θ postnorm shows that if we normalize the input signal f (t) = U (t; µ, K 0 ) to having unit contrast, then the corresponding post-normalized differential entity is approximately constant for temporal peaks with different temporal duration as determined by the parameter K 0 . In these respects, this time-causal scale selection method implies a good approximate transfer of the scale selection property of treating similar temporal structures of different temporal duration in a uniform manner.

Response to a localized Gaussian onset blob
Consider a local idealized spatio-temporal image pattern defined as the combination of a rotationally symmetric Gaussian blob g(x, y; s 0 ) over the spatial domain and a timecausal onset ramp t u=0 U (u; µ, K 0 ) du of the form (66) over the temporal domain Again defining the temporal scale-space representation of this spatio-temporal image pattern of the form (122) with the temporal scale-space kernel chosen as the composed kernel U (t; µ, K) corresponding to a cascade of first-order integrators with equal time constants coupled in cascade (120), it follows from the semi-group property of the spatial Gaussian kernel and the semi-group property (45) of the timecausal temporal kernel U (t; µ, K) that the spatio-temporal scale-space representation will be of the form L(x, y, t; s, τ ) = g(x, y; s 0 + s) t u=0 U (u; µ, K 0 ) du.  Table 8 Numerical estimates of the value ofK at which the temporal component of the determinant of the spatio-temporal Hessian assumes its maximum over temporal scale (with the discrete expression over discrete temporal scales extended to a continuous variation) as function of K 0 and for either (i) variance-based normalization with γ = 5/4, (iii) variance-based normalization with γ = 1 and (iv) Lp-normalization with p = 1.
For the case of variance-based normalization with γ = 5/4, (ii) the post-normalized magnitude measure L ζζ,maxmagn,norm γ=1 according to (20) and at the corresponding scale (i) is also shown. Note that for γ = 5/4 the temporal scale estimateK constitutes a good approximation to the temporal scale estimate being proportional to the temporal scale of the underlying temporal peak and that the maximum magnitude estimate θpostnorm| γ=1 constitutes a good approximation to the maximum magnitude measure being constant under variations of the temporal duration of the underlying spatio-temporal image structure.
The first-order temporal derivative of the spatial Laplacian of the Gaussian. For the scale-normalized differential entity ∂ t,norm (∇ 2 (x,y),norm L) constituting an idealized model of a "non-lagged" LGN cell [78, Figure 3(left)] will by a combination of the Laplacian response of Gaussian ∇ 2 g(x, y; s) = (x 2 +y 2 −2s)/s 2 g(x, y; s), the first-order temporal derivative L t (t; µ, K) of a time-causal onset ramp in Equation (68) and the temporal scale normalization operation L ζ (t; µ, K) = (µ √ K) γτ L t (t; µ, K) in Equation (53) assume the form ∂ t,norm (∇ 2 (x,y),norm L) Specifically, based on previously established scale selection properties of the spatial Laplacian of the Gaussian (Lindeberg [66,74]) and the first-order scale-normalized temporal derivatives of the time-causal scale space (Section 4.3), it follows that this spatio-temporal differential entity will for spatial and temporal scale normalization powers γ s = 1 and γ τ = 1/2, respectively, assume its local extremum over both spatial and temporal scales at spatial scalê and at a temporal scale that is a good approximation of the temporal scale of the temporal onset rampK ≈ K 0 corresponding to (see Table 2) Thus, simultaneous spatio-temporal scale selection using the differential entity ∂ t,norm (∇ 2 (x,y),norm L) applied to the model signal (143) will estimate both the spatial extent and the temporal duration of a Gaussian onset blob.
then by the general transformation property of the time-causal limit kernel under temporal scaling transformations by a temporal scaling factor S = c j that is an integer power of the distribution parameter c of the time-causal limit kernel (Lindeberg [78,Equation (44)]) S Ψ (S t; S 2 τ, c) = Ψ (t; τ, c), it holds that the corresponding spatio-temporal scale-space representations of two temporally scaled video sequences f (x , y , t) = f (x, y, t) for (x , y , t ) = (x, y, St) are related according to (Lindeberg [78,Equation (46)]) L (x , y ; , t ; s , τ , c) = L(x, y, t; s, τ, c) for (s , τ ) = (s, S 2 τ ) if S = c j . The corresponding scalenormalized temporal derivatives are in turn related according to (104) L ζ n (x , y , t ; s , τ , c) = S n(γτ −1) L ζ n (x, y, t; s, τ, c).
If the scale-normalized temporal derivative L ζ n (x, y, t; s, τ, c) computed from the original video sequence f assumes a local extremum over temporal scales at (x, y, t; s, τ ) = (x 0 , y 0 , t 0 ; s 0 , τ 0 ), then by the general scale-invariance property of temporal scale selection in the temporal scale-space concept based on the time-causal limit kernel, which is described in Section 5.3, it follows that the scale-normalized temporal derivative L ζ n (x , y , t ; s , τ , c) computed from the temporally scaled video sequence f will assume a local extremum over temporal scales at This scale-invariant property can also be extended to spatiotemporal derivatives L ξ m 1 η m 2 ζ n and spatio-temporal differential invariants D norm defined in terms of homogenous polynomials as well as homogeneous rational expressions of such scale-normalized spatio-temporal derivatives. In this way, by performing both the temporal modelling of the underlying temporal signal in terms of the time-causal limit kernel and using a temporal scale-space concept based on the timecausal limit kernels, we can support fully scale-covariant temporal scale estimates for temporal scale selection in video data defined over a time-causal spatio-temporal domain.
The only component that remains is to determine how the original temporal scale estimateτ depends on the distribution parameter c and the temporal scale normalization parameter γ τ for some value of τ 0 .

Temporal scale selection in spectro-temporal audio data
For audio signals, corresponding temporal scale selection methods can be applied to a time-causal spectro-temporal domain, with the 2-D spatial domain of video data over the spatial dimensions (x, y) conceptually replaced by a 1-D logspectral domain over the logspectral dimension ν in the spectrogram computed at any temporal moment using a timecausal receptive field model as proposed in (Lindeberg and Friberg [82,83]) or with the time-causal kernels in that model replaced by the time-causal limit kernel (Lindeberg [78]).
The analogous operations to the first-and second-order temporal derivatives of the spatial dimension would then be the first-and second-order temporal derivatives of the second-order derivative in the logspectral dimension ∂ t,norm (L νν,norm ) = s γs τ γτ /2 L ννt (154) where s denotes the logspectral scale and τ the temporal scale. By calibrating the logspectral scale normalization parameter γ s such that the selected temporal scale should reflect the logspectral width of a spectral band, it follows that we should use γ s = 3/4. By calibrating the temporal scale normalization parameter γ τ such that the selected temporal scale of ∂ t,norm (L ν,norm ) should reflect the temporal duration of an onset, if follows that we should use γ τ = 1/2 for this operator. By instead calibrating the temporal scale normalization parameter γ τ such that the selected temporal scale of ∂ tt,norm (L νν,norm ) should reflect the temporal duration of a beat, if follows that we should use γ τ = 3/4 for that operator.
Note that these operations can be expressed both over frequency-time separable spectro-temporal receptive fields and over glissando-adapted spectro-temporal receptive fields if we for glissando-adapted receptive fields also replace the temporal derivative operator ∂ t by the corresponding glissandoadapted temporal derivative operator ∂t = ∂ t + v ∂ ν , where v denotes the glissando parameter.

Summary and conclusions
In this treatment, we have proposed a new theoretical framework for temporal scale selection in a time-causal scalespace representation. Starting from a general survey of previously proposed temporal scale-space concepts and a detailed analysis of their relative advantages and disadvantages, we have focused our efforts on the time-causal scale-space concept based on first-order integrators coupled in cascade and analysed the extent to which scale-space properties that hold for the scale-invariant non-causal Gaussian temporal scale-space concept can be transferred to this time-causal scale-space concept. Specifically, we have analysed this timecausal scale-space concept for two specific ways of distributing the intermediate temporal scale levels using either (i) a uniform distribution over the temporal scales as parameterized by the variance of the temporal scale-space kernel and corresponding to convolution with temporal kernels that are Laguerre functions and in turn corresponding to temporal derivatives of the Gamma distribution or (ii) a logarithmic distribution taken to a recently proposed time-causal limit kernel with an infinitely dense distribution of temporal scale levels towards zero temporal scale.
For peak and ramp detection, we have shown that for the time-causal temporal scale space concept based on firstorder integrators with equal time constant coupled in cascade, we can reasonably well estimate the temporal scale of a localized temporal peak or a localized onset ramp with corresponding good approximation of constancy of appropriately post-normalized scale-normalized magnitude measures of the corresponding feature detectors under variations in the temporal duration of the underlying temporal peak or the underlying temporal ramp. For a non-localized sine wave signal, the lack of temporal scale invariance is, however, substantial both with regard to a systematic offset in temporal scale estimates and a lack of corresponding constancy of the magnitude measures over variations of the wavelength of the underlying sine wave.
For the time-causal temporal scale-space concept based on convolution with the time-causal limit kernel with an underlying logarithmic distribution of the temporal scale lev-els and taken to the limit of the time-causal limit kernel with an infinitely dense distribution of temporal scale levels near temporal scale zero, we have on the other hand shown that it is possible to achieve perfect temporal invariance in the respects that (i) the temporal scale estimates in dimension [time] are proportional to the wavelength of the underlying sine wave and (ii) the magnitude measures remain constant under variations of the wavelength of the sine wave.
Additionally, we have shown a general scale invariance result that holds for any temporal signal and which states that for temporal scaling transformation with a temporal scaling factor given as an integer power of the distribution parameter c of the time-causal limit kernel that is used for generating the temporal scale space, it holds that: In these respects, the proposed framework for temporal scale selection in the scale-space concept based on the time-causal limit kernel provides the necessary mechanisms to achieve temporal scale invariance while simultaneously being expressed over a time-causal and time-recursive temporal domain. From a theoretical perspective, this is a conceptually novel type of construction that has not previously been achieved based on any other type of time-causal temporal scalespace concept.
As experimental confirmation of the derived theoretical results regarding temporal scale selection properties, we have presented experimental results of applying two types of more specific temporal scale selection algorithms to onedimensional temporal signals, based on either (i) sparse scalespace extrema detection by detecting local extrema of feature responses over both time and temporal scales or (ii) dense feature maps over temporal scales here specifically manifested in terms of a temporal quasi quadrature entity that constitutes an energy measure of the local strength of scalenormalized first-and second-order temporal derivates.
We have also described practical details to handle in timecausal scale selection algorithms in relation to the inherent temporal delays of time-causal image measurements and proposed specific mechanisms to handle the differences in temporal delays between time-causal scale-space representations at different temporal scales.
Experimental results presented for synthetic and real onedimensional temporal signals show that it is possible to compute local estimates of temporal scale levels that in units of the standard deviation of the underlying temporal scalespace kernel are proportional to the temporal duration of the underlying structures in the temporal signal that gave rise to the filter responses.
Beyond these two specific ways of expressing temporal scale selection mechanisms, we argue that the theoretical results presented in the paper should also more generally open up for extensions to other ways of comparing timedependent filter responses at multiple temporal scales.
Experimental results obtained by applying this temporal scale selection theory to video analysis will be reported in a companion paper [79].
The non-causal temporal scale selection theory developed in this paper as a baseline and reference for time-causal temporal scale selection can of course also be used for analysing pre-recorded time-dependent signals in offline or time-delayed scenarios.
A Why a semi-group property over temporal scales leads to undesirable temporal dynamics in the presence of temporal delays One way of understanding why the assumption about a semigroup property over temporal scales may lead to undesirable temporal dynamics for a temporal scale space representation involving temporal delays can be obtained as follows: Ideally, for a temporal scale-space concept involving a temporal delay one would like the temporal delay δ to be proportional to the temporal scale parameter σ in terms of dimension [time] for some constant C > 0. For a temporal scale-space kernel with finite 4 temporal variance τ , this corresponds to letting 4 Regarding the assumption of a finite temporal variance, it is interesting to compare the situation with the time-causal semi-group kernel φ(t; τ ) = 1 √ 2π t 3/2 τ e −τ 2 /2t derived by Fagerström [20,Equation (27)] and Lindeberg [70,Equation (93)]. For this kernel, the firstand second-order temporal moments are not finite ∞ t=0 t φ(t; τ ) dt → ∞ and ∞ t=0 t 2 φ(t; τ ) dt → ∞, implying that the analysis in this appendix breaks down if applied to the time-causal semi-group, since this analysis is based on the additive properties of mean values and the temporal delay at any temporal scale be proportional to the square root of the temporal scale parameter τ according to Let us next assume that we have to two temporal scale-space kernels h(t; τ 1 , δ 1 ) and h(t; τ 2 , δ 2 ) with finite temporal variances τ 1 and τ 2 and finite temporal means δ 1 and δ 2 from the same family of temporal kernels h. If the temporal kernels are to obey a semi-group property over temporal scales, then by the additive property of mean values and variances under convolution of positive functions, it follows that the composed temporal scale-space kernel should be given by h(·; τ 1 , δ 1 ) * h(t; τ 2 , δ 2 ) = h(t; τ 1 + τ 2 , δ 1 + δ 2 ). (158) This property should for example hold for the non-causal Gaussian temporal scale-space kernels (2) if we require the kernels to obey a semi-group property over temporal scales.
Combining this property with a fixed relationship between the temporal delay δ and the temporal scale τ according to δ = φ(τ ) does, however, then lead to This implies that the function φ must be additive in terms of its argument τ , implying increasingly longer temporal delays at coarser temporal scales and thus a violation of the desirable form of temporal dynamics δ = φ(τ ) = C √ τ . We can, however, remedy the situation by replacing the temporal semi-group property with a weaker cascade smoothing property over temporal scales L(·; τ 2 , δ 2 ) = h(·; (τ 1 , δ 1 ) → (τ 2 , δ 2 )) * L(·; τ 1 , δ 1 ), variances for non-negative distributions. The fact that the first-and second-order temporal moments are infinite for the time-causal semigroup, does on the other hand also reflect undesirable temporal dynamics, since temporal smoothing with such a kernel leads to slow and smeared out temporal responses compared to temporal smoothing with a temporal kernel having finite first-and second-order temporal moments. If we measure the temporal delay of the time-causal semi-group kernel by the position of the temporal maximumt = τ 2 /3 (Lindeberg [70,Equation (119)]) and its temporal extent from the difference between the time instances at which the one-dimensional time-causal semi-group kernel assumes half its maximum value ∆t ≈ 0.900 τ 2 (Lindeberg [70,Equation (122)]), then the temporal delay and the temporal extent of the time-causal semi-group kernel are indeed proportional. Those measures of the temporal delay and the temporal extent of the temporal kernel are, however, not the same as used in the in the arguments in this appendix. The example with the time-causal semi-group therefore demonstrates that at the cost of infinite first-and second-order temporal moments it is possible to find a a temporal smoothing kernel that both obeys the semigroup property and a proportionality relation between measures of the temporal delay and the temporal extent in dimensions of [time]. Due to the infinite first-and second-order temporal moments, the temporal dynamics is, however, undesirable anyway.
Based on this form of cascade smoothing property over temporal scale, we can both (i) guarantee non-creation of new structures in the signal from finer to coarser temporal scales based on the scale-space properties of the temporal scalespace kernel g and (ii) achieve temporal delays that increase linearly with the temporal scale parameter in terms of dimension [time] such that δ 1 = C √ τ 1 and δ 2 = C √ τ 2 in (160). In our temporal scale-space concept based on truncated exponential kernels coupled in cascade (Lindeberg [57,77,78]; Lindeberg and Fagerström [81]), we can specifically note that (i) the special case when all the time constants are equal implies a semi-group property over discrete temporal scales and longer temporal delays at coarser temporal scales (see the second row in Figure 1) whereas the (ii) the special case with logarithmically distributed temporal scales implies that only a weaker cascade smoothing property holds and which enables much faster temporal response properties (see the third and fourth rows in Figure 1).
Since any time-causal temporal scale-space representation will give rise to non-zero temporal delays, and we have shown in this section how the assumption of a semi-group structure over temporal scale leads to undesirable temporal dynamics in the presence of temporal delays, we argue that one should not require a semi-group structure over temporal scales for time-causal scale space and instead require a less restrictive cascade smoothing property over temporal scales.

B Scale normalization of temporal derivatives in Koenderink's scale-time model
In his scale-time model, Koenderink [46] proposed to perform a logarithmic mapping of the past via a time delay and then applied Gaussian smoothing in the transformed temporal domain. Following the slight modification of this model proposed in (Lindeberg [78,Appendix 2]) to have the temporal kernels normalized to unit L 1 -norm such that (165) Table 9 Regular first-, second-and third-order temporal derivatives of Koenderink's scale-time kernel (renormalized according to (172)).
Scale-normalized temporal derivatives of Koenderink's scale-time kernel: Using variance-based normalization, the corresponding scale-normalized temporal derivatives are for a general value of γ given by which for the specific value of γ = 1 reduce to or when using Lp-normalization for p = 1:  Table 10 Scale-normalized first-and second-order temporal derivatives of Koenderink's scale-time kernel using either variance-based normalization for a general value of γ, variance-based normalization for γ = 1 or L P -normalization for p = 1.
a constant signal should remain unchanged under temporal smoothing, these kernels can be written on the form where δ represents the temporal delay and σ is a dimensionless temporal scale parameter relative to the logarithmically transformed temporal domain. The temporal mean of this kernel is (Lindeberg [ Based on these characteristics, we can define scale-normalized temporal derivatives of Koenderink's scale-time kernel according to Equations (166)-(171) in Table 10.
under the conditions c > 1 and σ < √ log 2 ≈ 0.832 by requiring the first-and second-order temporal moments of the kernels in the two families to be equal.
Given this approximate mapping between the time-causal limit kernel and the temporal kernels in Koenderink's scaletime model, we can approximate the positions of the temporal peak, the peaks in the first-and second-order temporal derivatives of the time-causal limit kernel based on our previously derived expressions for the maximum point t max , the inflection points t inf lect1 and t inf lect2 as well as the zero-crossings of the third-order derivative t 3,1 , t 3,2 and t 3,2 according to (178)-(183) in Appendix B: Specifically, this leads to the following estimates of how the temporal width of the first-and second-order temporal derivatives depend on the distribution parameter c which can be compared to the corresponding width measures for the non-causal Gaussian kernel (196) Figure 15 shows graphs of the width measures (193) and (194) of the first-and second-order temporal derivatives time-causal limit kernel obtained as obtained from a scaletime approximation. As can be seen from the graphs, the width measures vary by about 30 % when the distribution parameter is varied between c = √ 2 and c = 2. Thus, the value of the distribution parameter c must be taken into explicit account when transferring the temporal scale parameter τ to a characteristic length estimate on the temporal axis. Notably when the distribution parameter tends to c → 1, the temporal width estimates approach the corresponding width estimates (195) and (196) of the Gaussian kernel.
Given that a temporal feature has been detected from a local maximum over temporal scales in either the firstor second-order temporal derivative of the time-causal limit kernel, if we use the behaviour of the Gaussian temporal scale-space model (195) and (196) for additional calibration of the proportionality constant of the scale estimate, we do then obtain the following estimatesd of the temporal duration of the corresponding temporal feature as function of the temporal scale estimateτ and the distribution parameter c: √τ .