On the distribution theory of over-dispersion

Abstract

An overview of the evolution of probability models for over-dispersion is given looking at their origins, motivation, first main contributions, important milestones and applications. A specific class of models called the Waring and generalized Waring models will be a focal point. Their advantages relative to other classes of models and how they can be adapted to handle multivariate data and temporally evolving data will be highlighted.

1. Introduction

Data analysts have often to deal with data that exhibit a variability that differs from what they expect on the basis of the hypothesized model. The phenomenon is known as overdispersion if the observed variability exceeds the expected variability or underdispersion if it is lower than expected.

Such differences between observed and nominal variances can be interpreted as brought about by failures of some of the basic assumptions of the model. These can be classified by the mechanism leading to them. As summarized by Xekalaki ([2006]), in traditional experimental contexts, they may be caused by deviations from the hypothesized structure of the population, due to lack of independence between individual item responses, contagion, clustering, and heterogeneity. In observational study contexts, on the other hand, they are the result of the method of ascertainment, which can lead to partial distortion of the observations. In both contexts, the observed value x no longer represents an observation on the original variable X, but constitutes an observation on a random variable Y whose distribution (the observed distribution) is a distorted version of the distribution of X (original distribution).

Such practical situations have been noticed since over a century ago (e.g. Lexis [1879]; Student [1919]). The Lexis ratio appears to be the first statistic suggested for testing for the presence of over- or under-dispersion relative to a binomial hypothesized model in populations structured in clusters. Also, for count data, Fisher ([1950]) considered using the sample index of dispersion for testing the appropriateness of a Poisson distribution for an observed variable Y.

The paper is structured as follows. Section 2 introduces the reader to the various approaches to modelling overdispersion in the case of traditional experimental contexts. Section 3 highlights approaches in the case of observational study contexts. Section 4 focuses on the case of heterogeneous populations followed by Sections 5 and 6, which look into a particular type of distribution, the generalized Waring distribution, and its relevance in the context of applications under the various scenaria leading to over-dispersion mentioned above. Through the prism of these scenaria, a bivariate version of it is also presented, and its use in applied contexts is discussed in Section 7. A multivariate version of it is also given, and its application potential is outlined in Section 8. Finally, Sections 9 and 10 present a model for temporally evolving data, the multivariate generalized Waring process, and an application illustrating its practical potential.

As the field of accident studies has received much attention, and various theories have been developed for the interpretation of factors underlying an accident situation, most of the models will be presented in accident or actuarial data analysis contexts. Of course the results can be adapted in a great variety of situations with appropriate parameter interpretations so that they can be applied in several other fields ranging from economics, inventory control and insurance through to demometry, biometry, psychometry and web access modeling, as the case is with the application discussed in Section 10.

2. Modelling over - or under - dispersion in traditional experimental contexts

One important, but often ignored by data analysts, implication of using single parameter distributions such as the Poisson distribution to analyse data is that the variance can be determined by the mean, a relation that collapses by the presence of overdispersion. If this is ignored in practice, any form of statistical inference may induce low efficiency, although, for modest amounts of overdispersion this may not be the case (Cox [1983]). So, insight into the mechanisms that induce over (or under) dispersion is required when dealing with such data. Such insight can be gained by looking at the above-mentioned potential triggering sources as classified by Xekalaki ([2006]).

2.1 Lack of independence between individual responses

In accident study related contexts, where one is interested in the total number of reported accidents $Y={\displaystyle \sum _{i=1}^n{Y}_i}$ in a total number of accidents, n, that actually occurred, when accidents are reported with equal probabilities p = P(Y _i  = 1) = 1 − P(Y _i  = 0), but not independently (Cor(Y _i , Y _j ) = ρ ≠ 0), the mean of Y will still be E(Y) = np, but its variance will be $V\left(Y\right)=V\left({\displaystyle \sum _{i=1}^n{Y}_i}\right)=\mathit{np}\left(1-p\right)+2\left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right)\mathit{\rho p}\left(1-p\right)=\mathit{np}\left(1-p\right)\left(1+\rho \left(n-1\right)\right)$, which exceeds that anticipated under a hypothesized independent trial binomial model if ρ > 0 (over-dispersion) and is exceeded by it if ρ < 0 (under-dispersion).

2.2 Contagion

Another common reason for a variance differing from what is anticipated, is that when the assumption that the probability of the occurrence of an event in a very short interval is constant fails. This framework is the classical contagion model (Greenwood and Yule [1920]; Xekalaki [1983a]).

In data modelling problems faced by actuaries, for example, this model postulates that initially all individuals have the same probability of incurring an accident, but later this probability changes by each accident sustained. It is assumed, specifically, that none of the individuals has had an accident (e.g. new drivers or persons who are just beginning a new type of work), but later the probability with which a person with Y = y accidents by time t will have another accident in the time period from t to t + dt is of the form (k + my)dt. This leads to the negative binomial as the distribution of Y with p.f. $P\left(Y=y\right)=\left(\begin{array}{c}\hfill k/m\hfill \\ \hfill y\hfill \end{array}\right)e^{-\mathit{kt}}{\left(1-e^{-\mathit{mt}}\right)}^y$ with μ = E(Y) = k(e^mt − 1)/m, and V(Y) = ke^mt(e^mt − 1)/m = μe^mt.

2.3 Clustering

A frequently overlooked clustered structure of the population may also induce over - or under - dispersion.

In an accident context again, an accident is regarded as a cluster of injuries:

The number Y of injuries incurred by persons involved in N accidents can naturally be thought of as expressed by the sum Y = Y₁ + Y₂ + … + Y _N of the numbers Y _i of injuries resulting from the i ‐ th accident, assumed to be i.i.d. independently of the total number of accidents N, with mean μ and variance σ². In this case, $E\left(Y\right)=E\left({\displaystyle \sum _{i=1}^N{Y}_i}\right)=\mathit{\mu E}\left(N\right)$ and $V\left(Y\right)=V\left({\displaystyle \sum _{i=1}^N{Y}_i}\right)={\sigma }^{2}E\left(N\right)+{\mu }^{2}V\left(N\right)$.

So, when N is a Poisson variable with mean E(N) = θ = V(N), the last relationship leads to overdispersion or underdispersion according as σ² + μ² is greater or less than 1.

The first such model was introduced by Cresswell and Froggatt ([1963]) in a different accident context whereby each person is liable to spells of weak performance during which all of the person’s accidents occur. So, if the number N of spells in a unit time period is Poisson distributed with mean θ, and within spells a person can have 0 accidents with probability 1 − m log p, m > 1/log p, 0 < p < 1 and n accidents (n ≥ 1) with probability m(1 − p)ⁿ/n, m, n > 0 the observed distribution of accidents is the negative binomial distribution with probability function $P\left(Y=y\right)=\left(\begin{array}{c}\hfill \mathit{\theta m}+y-1\hfill \\ \hfill y\hfill \end{array}\right)p^{\mathit{\theta m}}{\left(1-p\right)}^y$. This model, known in the literature as the spells model, can also lead to other forms of overdispersed distributions (e.g. Xekalaki [1983a], [1984a]).

2.4 Heterogeneity

Assuming a homogeneous population when in fact the population is heterogeneous, i.e., when its individuals have constant, but unequal probabilities of sustaining an event can also lead to overdispersion. In this case, each member of the population has its own value of the parameter θ and probability density function f(⋅ ; θ).

So, with θ regarded as the inhomogeneity parameter and varying from individual to individual according to any continuous, discrete, or finite step distribution G(⋅) of mean μ and variance σ², one is led to an observed distribution for Y with probability density function f _Y (y) = E _G (f(y; θ)) = ∫  _Θ f(y; θ)dG(θ), where Θ is the parameter space. Models of this type are known as mixtures. (For details on their application in the statistical literature see e.g. Karlis and Xekalaki [2003]; McLachlan and Peel [2001]; Titterington [1990]). Under such models, the variance of Y consists of two additive components, one representing the variance part due to the variability of θ and one due to the inherent variability of Y if θ did not vary, i.e., V(Y) = V(E(Y|θ)) + E(V(Y|θ)). This offers an explanation as to why mixture models are often referred to as overdispersion models.

It should be noted that a similar idea forms the basis for analysis-of-variance (ANOVA) models, where the total variability can be split into additive components, the ‘between groups’ and the ‘within groups’ components. In the case of the Poisson (θ) distribution, we have in particular that V(Y) = E(θ) + V(θ). Based on the fact that in this case, the factorial moments of Y coincide with the moments of θ about the origin, Carriere ([1993]) proposed a test of the hypothesis that a Poisson mixture fits a data set.

Mixed Poisson distributions were first introduced by Greenwood and Woods ([1919]) in the context of accident studies. Assuming that an individual’s accident experience Y|θ is Poisson distributed with parameter θ that was varying from individual to individual according to a gamma distribution with mean μ and index parameter μ/γ, they obtained a negative binomial distribution for Y with probability function $P\left(Y=y\right)=\left(\begin{array}{c}\hfill \mu /\gamma +y-1\hfill \\ \hfill y\hfill \end{array}\right){\left\{\gamma /\left(1+\gamma \right)\right\}}^y{\left(1+\gamma \right)}^{-\mu /\gamma }$ and with mean and variance given respectively by E(Y) = μ and V(Y) = μ(1 + γ), where γ represents the over-dispersion parameter.

The mixed Poisson process has been popularised in the actuarial literature by Dubourdieu ([1938]) gamma mixed case was treated by Thyrion ([1969]).

Numerous other mixtures have since then been proposed in the literature for interpreting overdispersion in data, such as binomial mixtures (e.g. Tripathi et al. [1994]), negative binomial mixtures (e.g., Xekalaki [1983a], [c], [1984a]; Irwin [1975]), normal mixtures (e.g. Andrews and Mallows [1974]) and exponential mixtures (e.g. Jewell [1982]). Discrete Poisson mixtures with finite step distributions for the Poisson parameter θ have also been proposed, the interest being on creating clusters of data by grouping the observations on Y according to some criterion (cluster analysis). The number of clusters can be decided on the basis of a testing procedure for the number of components in the finite mixture (Karlis and Xekalaki [1999]).

2.4.1 Heterogeneity in mixture models treating the parameter θ as the dependent variable in a regression model

Heterogeneity in models with explanatory variables can be modelled, by assuming that Y has a parameter θ varying from individual to individual according to some regression model θ = η(x; β) + ε, where x is a vector of explanatory variables, β is a vector of regression coefficients, η is a function of a known form and ε has some known distribution. Such models are known in the literature as random effect models and have been extensively studied within the broad family of Generalized Linear Models. As a simple example in the case of a single covariate, say X, consider data Y _i  , i = 1, 2, … , n coming from a Poisson population with mean θ determined by log θ = α + βx + ε for some constants α, β and with ε having a distribution with mean 0 and variance say ϕ. In this case, the marginal distribution of Y is no longer the Poisson distribution. It is a mixed Poisson distribution, with some mixing distribution g(⋅) clearly depending on the distribution of ε. In particular, $Y\sim \mathit{Poisson}\left(t{e}^{\left(\alpha +\mathit{\beta x}\right)}\right)\underset{t}{\wedge }g\left(t\right)$ where t = e^ε.

Negative Binomial and Poisson Inverse Gaussian regression models have also been proposed as overdispersed alternatives to the Poisson regression model (e.g. Lawless [1987]; Dean et al. [1989]; Xue and Deddens [1992]). The case of a two finite step distribution, the finite Poison mixture regression model of Wang et al.’s ([1996]) results. The similarity of the mixture representation and the random effects one is discussed in Hinde and Demetrio ([1998]).

In meta-analysis contexts, overdispersion (or underdispersion) refers to variance inflation (or deflation) relative to that anticipated by the fixed effects model. Two possible causes of such phenomena are a population structure in clusters or mixing resulting in a compound distribution. Kulinskaya and Olkin ([2014]) proposed approaching the problem of specification of a random effects model in meta-analysis in terms of a multiplicative model for the distribution of the effect size parameters that allows inflation or deflation. The model considered was motivated by overdispersion induced by intra-class correlation in the model assumed for the distribution of the i-th effect size estimate. In particular, the variance of the estimator ${\widehat{\theta }}_i$ of the effect size parameter θ _i in the i-th study is assumed to be of the form ${\sigma }_{{\widehat{\theta }}_i}^2=\left(1+\alpha \left(n_i\right)\gamma \right){\sigma }_i^2$, where α(n _i ) are some known functions of the sample sizes $n_i,{\sigma }_i^2$ is the within the i-th study variance, i = 1, 2, …, k and γ is interpreted as an intra class correlation parameter.

2.4.2 Estimation and testing for overdispersion under mixture models

The structure of mixture models, including random effect models, entails different forms of variance-to-mean relationships. So, viewing the mean and variance of Y as represented by E(Y) = μ(β), and V(Y) = σ²(μ(β), λ) respectively for some parameters β, λ a number of estimation approaches have been proposed in the literature based on moment methods (e.g. Breslow [1990]; Lawless [1987]; Moore [1986]) and quasi or pseudo likelihood methods (e.g. Davidian and Carroll [1988]; McCullagh and Nelder [1989]; Nelder and Pregibon [1987]). The above representation for the mean and variance of Y allows also estimation in the case of multiplicative overdispersion as in McCullagh and Nelder ([1989]).

Testing for the presence of overdispersion or underdispersion, on the other hand, can be done by means of asymptotic arguments. Let f(y; θ) denote the density function of a random variable Y in the initial model. Cox ([1983]) showed that, under regularity conditions, the density of y in the overdispersed model, f _Y (y), admits a representation of the form $f_Y\left(y\right)=E_{\Theta }\left(f\left(y;\theta \right)\right)=f\left(y;{\mu }_{\theta }\right)+\frac{1}{2}{\sigma }_{\theta }^2\frac{{\partial }^{2}f\left(y;{\mu }_{\theta }\right)}{\partial {\mu }_{\theta }^2}+{\rm O}\left(1/n\right)$, with ${\mu }_{\theta }={\rm E}\left(\theta \right),{\sigma }_{\theta }^2=V\left(\theta \right)$ and Θ is the parameter space. This in turn implies that f _Y (y) can be put in the form f(y; μ _θ )(1 + εh(y, ϕ _θ )), where $h\left(y,{\varphi }_{\theta }\right)={\left[\frac{\partial logf\left(y;{\mu }_{\theta }\right)}{\partial {\mu }_{\theta }}\right]}^2+\frac{{\partial }^{2}logf\left(y;{\mu }_{\theta }\right)}{\partial {\mu }_{\theta }^2}$.

This representation entails overdispersion if ε > 0, underdispersion if ε < 0 and, of course, none of these complications if ε = 0. Cox ([1983]) suggested a testing procedure for the hypothesis ε = 0, which can be regarded as a general version of standard dispersion tests.

2.5 Zero adjusted models

It would be interesting to note that another aspect of the population structure that is often responsible for the phenomenon of over-dispersion or under-dispersion is the presence of an excess or a scant number of zeros. Though the models discussed in Sections 2.3 and 2.4 may capture over-dispersion or under-dispersion rather well, they cannot capture excess or scarcity of zeros. In the literature, this question has been addressed by two types of models known as zero-inflated (or zero-deflated) models, and hurdle models. A unified representation of the models is provided by f(y; ω) = ωI_{0}(y) + (1 − ω)f _Y (y), where Y is the count variable, I_{0}(⋅) is the indicator function and ω is a constant, whose values, if in (0,1) render a hurdle model for f _Y (0) = 0, a zero-inflated model for f _Y (0) ≠ 0, while negative values of it render a zero-deflated model.

Obviously, ω can be interpreted as the proportion of excess zeros in the case of the first two models and the above representation explains why there can be regarded as having a dual nature. They are (finite) mixtures, which account for heterogeneity, while at the same time, they are capturing a population structure in two clusters. However, in the case ω < 0 (zero-deflation), the model ceases to admit a mixture interpretation.

Zero-inflated and hurdle models have mostly been used for Poisson, generalized Poisson or negative binomial count distributions in various contexts (e.g. Ridout et al. [2001]; Gupta et al. [2004]; Famoye and Singh [2006]). Gupta et al. ([1996]) proposed a zero-adjusted generalized Poisson distribution and studied the effect of not using an adjusted model for zero-inflation or -deflation when the occurrence of zeroes differs from the anticipated one. Reviews of such models can be found in Ridout et al. ([1998]), Gschlößl and Czado ([2008]) and Ngatchou-Wandji and Paris ([2011]).

3. Over– or under–dispersion in observational study contexts - the effect of the method of ascertainment

Often, in connection with data collection based on observation or on recording values as produced by nature, the original distribution may not be reproduced due to various reasons. These may lead to partial destruction or partial enhancement (augmentation) of observations. The models that have been introduced to deal with such situations are respectively known as damage models introduced by Rao ([1963]) and generating models introduced by Panaretos ([1983]). The distortion mechanism is usually assumed to be manifested through the conditional distribution of the resulting random variable Y given the value of the original random variable X. Hence, the resulting (observed) distribution is a distorted version of the original distribution that can be represented as a mixture of the distortion mechanism. In particular, in the case of damage, $P\left(Y=r\right)={\displaystyle \sum _{n=r}^{\infty }P\left(Y=r|X=n\right)P\left(X=n\right)},r=0,1,2,\dots$, while, in the case of enhancement, $P\left(Y=r\right)={\displaystyle \sum _{n=1}^{r}P\left(Y=r|X=n\right)P\left(X=n\right)},r=1,2,\dots$.

Various forms of distributions have been considered for the distortion mechanism in the above two cases. In the case of damage, the most popular forms have been the binomial distribution Rao ([1963]), mixtures on p of the binomial distribution (e.g. Panaretos [1982]; Xekalaki and Panaretos [1983]) whenever damage can be regarded as additive (Y = X − U, U independent of Y) or in terms of the uniform distribution in (0, x) (e.g. Dimaki and Xekalaki [1990], [1996]; Xekalaki [1984b]) whenever damage can be regarded as multiplicative (Y = [RX], R independent of X and uniformly distributed in (0, 1)). The latter case has also been considered in the context of continuous distributions by Krishnaji ([1970]). The generating model was introduced and studied by Panaretos ([1983]).

Both, the generating model and the damage model offer a perceptive approach in actuarial contexts where one is interested in modelling the distributions of the numbers of accidents, of the damage claims, and of the claimed amounts. These models become relevant due to the fact that people have in general a tendency to under report their accidents, so that the reported (observed) number Y is less than or equal to the actual number X (Y ≤ X), but tend to over report damages incurred by them, so that the reported damage Y is greater than or equal to the true damage X (Y ≥ X).

Another type of distortion is induced by the adoption of a sampling scheme that assigns to the units in the original distribution unequal probabilities of inclusion in the sample. As a result, the value x of X is observed with a frequency that noticeably differs from that anticipated under the original density function f _X (x; θ). It represents an observation on a random variable Y whose probability distribution is the results of adjusting the probabilities of the anticipated distribution through weighting them with the probability with which the value x of X is included in the sample. So, if this probability is proportional to some weight function, w(x, β), β ∈ R, the recorded value x is a value of Y having density function f _Y (x; θ, β) = w(x; β)f _x (x; θ)/E(w(X; β)).

Distributions of this type are known as weighted distributions ( see, e.g. Cox [1962]; Fisher [1934]; Patil and Ord [1976]; Rao [1985]). For w(x; β) = x, these are known as size biased distributions. In actuarial data modelling contexts again, the weight function can represent reporting bias. In the context of reporting accidents or placing damage claims, for example, it can have a value that is directly or inversely analogous to the size x of X, the actual number of incurred accidents or the actual size of the incurred damage. The functions w(x; β) = x and w(x; β) = β^x (β > 1 or β < 1) are plausible choices. So, for example, in the case of a Poisson (θ) distributed X, these lead to distributions for Y that are of Poisson type. In particular, the weight function w(x; β) = x leads to a shifted Poisson distribution with probability function P(Y = x) = e^− θθ^x − 1/(x − 1) !, x = 1, 2, …, while the choice w(x; β) = β^x leads to a Poisson distribution P(Y = x) = e^− θβ(θβ)^x/x !, x = 0, 1, …. The value of the variance of the observed variable Y under the first assumption for w(x; β) is 1 + θ and exceeds that of X (overdispersion), while under the second assumption it is θβ implying overdispersion for β > 1 or underdispersion for β < 1.

4. Looking closer into the case of heterogeneity

Assuming a specific form for the distribution of the population that generated a data set implies that the mean to variance relation is given for this distribution, e.g. the Poisson distribution with a mean to variance ratio equal to unity. As has become obvious from the above, this relationship ceases to hold in real data sets however. This being rarely the case, flexible families have been sought in the literature by allowing the parameter θ of the original distribution to vary according to a distribution with probability density function, say g(⋅).

As mentioned before, a density function f _X (⋅) is a mixture on the parameter θ of the distribution function f(⋅ ; θ) with some mixing distribution G _θ (⋅), which can be continuous, discrete or a finite step distribution, if it can be written in the form f _X (x) = E _G (f(x; θ)) = ∫ _Θ  f(x; θ)dG(θ), where Θ is the parameter space. An appropriate choice of a mixing distribution allows its parameter to vary and acts as a means of “loosening” the structure of the initial model, thus offering more realistic interpretations of the mechanisms that generated the data.

A large number of Poisson mixtures have been developed. (For an extensive review, see Karlis and Xekalaki [2003], [2005]). The derivation of the negative binomial distribution, as a mixture of the Poisson distribution with a gamma distribution as the mixing distribution, originally obtained by Greenwood and Yule ([1920]) constitutes a typical example. Mixtures of the negative binomial distribution have also been widely used in connection with applications in a plethora of fields. These include the Yule distribution (Yule [1924]; Irwin [1941]; Xekalaki [1983c], [1984b]) the Waring distribution (Irwin [1963]) and the generalized Waring distribution (Irwin [1968], [1975]; Xekalaki [1981], [1983a], [1984a]), which contains the Yule distribution and the Waring distribution as a special cases.

In what follows, we focus on the generalized Waring distribution and its relevance in accident data modeling contexts.

5. The generalized Waring distribution

This was introduced by Irwin ([1968]) in connection to biological data and later was shown by him to arise as an accident distribution (Irwin [1975]). It is the distribution with probability generating function given by

$$G\left(s\right)=\frac{{\rho }_{\left(k\right)}}{{\left(a+\rho \right)}_{\left(k\right)}}F_1^2\left(a,k;a+k+\rho ;s\right),\alpha ,k,\rho >0$$

with ₂F₁(a, b; c; z) denoting the Gauss hypergeometric function ${\displaystyle {\sum }_{r=a}^x\left\{a_{\left(r\right)}{b}_{\left(r\right)}{z}^r\right\}/\left\{c_{\left(r\right)}r!\right\}}$, where h_(l) = Γ(h + l)/Γ(h), h > 0, l ∈ R.

Irwin’s starting point was Waring’s expansion (hence the distribution’s name) given by $\frac{1}{x-a}={\displaystyle \sum _{r=0}^{\infty }\frac{a_{\left(r\right)}}{x_{\left(r+1\right)}}}$, which he then generalized to $\frac{1}{{\left(x-a\right)}_{\left(k\right)}}={\displaystyle \sum _{r=0}^{\infty }\frac{a_{\left(r\right)}{k}_{\left(r\right)}}{x_{\left(k+r\right)}}\frac{1}{r!}},\alpha ,k>0$.

Hence, by multiplying both sides by ρ_(k), where ρ = x − a > 0, the successive terms of the resulting series could he regarded as defining a probability function, which he termed the generalized Waring distribution with parameters α, k, ρ. In particular, the probability function of the generalized Waring distribution with parameters α, k, ρ is given by

$$p_r=\frac{{\rho }_{\left(k\right)}}{{\left(a+\rho \right)}_{\left(k\right)}}\frac{a_{\left(r\right)}{k}_{\left(r\right)}}{{\left(a+k+\rho \right)}_{\left(r\right)}}\frac{1}{r!},\alpha ,k,\rho >0,r=0,1,2,\dots$$

where h_(l) = Γ(h + l)/Γ(h).

Notwithstanding the complexity of its structure, this distribution was shown to offer an insightful tool in the interpretation of accident data as will be seen below. Among its aspects that can be of practical value, is that, as shown by Xekalaki ([1983b]), it is a discrete self-decomposable distribution in Steutel and van Harn’s ([1979]) sense, hence infinitely divisible, implying that its probability generating function can be put in the form $G\left(s\right)=exp\left\{-\lambda {\displaystyle {\int }_s^1\frac{1-g\left(u\right)}{1-u}\mathit{du}}\right\}$, where λ = p₁/p₀ and g(⋅) denotes the probability generating function of the distribution with probability function satisfying the recurrence relation

$$q_n=\lambda \left\{n\frac{\mathit{ak}+\rho \left(a+k+\rho \right)}{\mathit{ak}\left(a+k+\rho +n\right)}-{\displaystyle \sum _{j=0}^{n-1}{q}_j}\left(\begin{array}{c}\hfill n\hfill \\ \hfill j\hfill \end{array}\right)\left(\begin{array}{c}\hfill a+k+\rho +n-1\hfill \\ \hfill j\hfill \end{array}\right)/\left[\left(\begin{array}{c}\hfill a+n-1\hfill \\ \hfill j\hfill \end{array}\right)\left(\begin{array}{c}\hfill k+n-1\hfill \\ \hfill j\hfill \end{array}\right)\right]\right\}$$

6. The generalized Waring distribution in relation to accident theory

The hypotheses that have formed the basis of investigations into the occurrence of accidents since almost a century ago are

• (i) Pure chance , giving rise to the Poisson distribution

• (ii) True contagion , i.e. the hypothesis that initially all individuals have the same probability of incurring an accident but that this probability is modified by each accident sustained.

• (iii) Apparent contagion (heterogeneity) , i.e. the hypothesis that individuals have constant but unequal probabilities of having an accident - the resultant distribution being a compound Poisson distribution (“accident proneness” model).

• (iv) The “Spells” Model , i.e each person is liable to periods of time during which the person’s performance is weak (spells). All of the person’s accidents occur within those spells. The numbers of accidents within different spells are independent and independent of the number of spells.

As already seen, the negative binomial distribution can be given a an accident proneness and a “spells” interpretation in the context of accident theory in terms of a gamma mixed Poisson distribution and a Poisson distribution generalized by a logarithmic distribution (Kemp [1967]).

Therefore, a good fit of the negative binomial is no help at all in distinguishing among the “proneness”, “contagion” and “spells” hypotheses. This is known as the discrimination problem between the compounded, contagion and generalized models for the negative binomial distribution and has been discussed by Arbous and Kerrich ([1951]); Bates and Neyman ([1952]); Gurland ([1959]) and Cane ([1974], [1977]). For an extensive bibliography on the accident hypotheses mentioned, see Kemp ([1970]).

6.1 Irwin’s “Proneness” model

As evident, in all three of the above models, the data are treated as if the individuals under observation were exposed to equal environmental risk, a fact criticized by Irwin ([1968]), who suggested a three-parameter distribution, which he called the “univariate generalized Waring distribution” (UGWD). He derived this distribution in a framework that allows separately for random factors, differences in the exposure of individuals to external risk of accident, and differences in proneness.

In particular, his model assumes a non homogeneous population with respect to personal and environmental attributes affecting the occurrence of accidents.

Let the distribution of the number, X, of accidents for individuals of equal proneness ν, and of equal exposure to external risk of accident λ|ν, i.e. λ for given ν), have probability generating function

$$G_{X|\lambda }\left(s\right)=exp\left\{\left(\lambda |\nu \right)\left(s-1\right)\right\}$$

in a unit time interval (0, 1). If the distributions of λ|ν and ν in the population at risk can be described by the probability density functions (pdf)

$$\left\{{\nu }^{-k}exp\left(-\lambda /\nu \right){\lambda }^{k-1}\right\}/\Gamma \left(k\right),v,k>0$$

and

$$\left\{\mathrm{\Gamma }\left(a+\rho \right){\nu }^{a-1}{\left(1+\nu \right)}^{-\left(a+\rho \right)}\right\}/\left\{\mathrm{\Gamma }\left(\rho \right)\mathrm{\Gamma }\left(a\right)\right\},a,\rho >0$$

respectively, the pgf of the resulting distribution of accidents will be {ρ_(k)2 F₁(a, k; a + k + ρ; s}/(a + ρ)_(k), i.e. the univariate generalized Waring distribution with parameters a, k and ρ, which will be denoted by UGWD(a, k; ρ). Here, ₂ F₁(a, b; c; z) denotes the Gauss hypergeometric function ${\displaystyle {\sum }_{r=a}^x\left\{a_{\left(r\right)}{b}_{\left(r\right)}{z}^r\right\}/\left\{c_{\left(r\right)}r!\right\}}$, where h_(l) = Γ(h + l)/Γ(h), h > 0, l ∈ R. For more information about the UGWD the reader is referred to the work of Irwin ([1963], [1968], 1975); Xekalaki ([1981]) and the references therein and Xekalaki ([1983a]).

6.2 The “Contagion” model

Xekalaki ([1983a]), extended the assumptions of the classical contagion model developed by Greenwood and Yule ([1920]) by considering a population of individuals exposed to varying accident risk.

In particular, assume that at time t = 0 none of the individuals has had an accident. This would be true if, for example, with a population of new drivers or of individuals just beginning a new type of work. Suppose that during the time period from t to t + dt a person with x accidents by time t can incur another accident with a probability of {(k + x)/(1 + λt)}λdt (independent of the times of the previous accidents), where k is a positive constant and λ refers to the individual’s risk exposure. At t = 0, since x = 0, the probability of an accident is kλdt. Hence, what the model basically assumes is that, initially, the probability of having an accident is not the same for each individual, but depends on the external conditions; later, the probability is also affected by the number of preceding accidents. Under these assumptions and if differences in the exposure to accident risk can be thought of as governed by a distribution with probability density function given by {Γ(a + ρ)v^a − 1(1 + ν)^{− (a + ρ)}}/{Γ(ρ)Γ(a)}, the final distribution of accidents over a unit period of time turns out to be UGWD(a, k; ρ).

The above derivation of the generalized Waring distribution closely relates to a modeling approach whereby the distribution of accident occurrences in a time internal (0, t) is regarded as underpinned by a stochastic process and, in particular, by a pure birth process {X _t , t = 0, 1, 2, …} where the probability of a person to incur an accident in (t, t + dt), having had x accidents by time t is P(X_t + δt = x + 1|X _t  = x) = f _λ (n, t)δt + o(δt).

Irwin ([1941]), followed later by Arbous and Kerrich ([1951]), derived the negative binomial distribution on the hypothesis solving the associated Kolmogorov forward differential equations by a method due to McKendrick ([1925]). Specifically, assuming that individuals can have during the time period from t to dt, individuals can have 0 accidents with probability 1 − f _λ (x, t)dt, 1 accident with probability f _λ (x, t)dt and > 1 accidents with probability 0, he solved the resulting system of Kolmogorov forward difference-differential equations

$$\begin{array}{l}\frac{\partial }{\partial t}{P}_{\lambda }\left(0,t\right)=-f_{\lambda }\left(0,t\right)P_{\lambda }\left(0,t\right)\\ \frac{\partial }{\partial t}{P}_{\lambda }\left(x,t\right)=-f_{\lambda }\left(x,t\right)P_{\lambda }\left(x,t\right)+f_{\lambda }\left(x-1,t\right)P_{\lambda }\left(x-1,t\right),x\ge 1\end{array}$$

in terms of a single difference-differential equation involving the probability generating function G _λ (s; t) of X _t given by

$$\frac{\partial }{\partial t}{G}_{\lambda }\left(s;t\right)=\left(s-1\right){\displaystyle \sum _{x=0}^{\infty }{s}^x{f}_{\lambda }\left(x,t\right)P_{\lambda }\left(x,t\right)}$$

where $G_{\lambda }\left(s;t\right)={\displaystyle \sum _{x=0}^{\infty }{P}_{\lambda }\left(x,t\right)s^x}$. (He obtained this equation by multiplying the i-th equation of the system by s^i − 1, i = 1, 2, … and summing the resulting equations).

Assuming further that f _λ (x, t) = λ(k + mx), k, m > 0 and subject to the initial conditions G _λ (1; t) = G _λ (s; 0) = 1, he obtained for the distribution of accidents

$$G_{\lambda }\left(s;t\right)={\left[e^{\mathit{\lambda mt}}-s\left(e^{\mathit{\lambda mt}}-1\right)\right]}^{-k/m}\text{,}$$

i.e. the probability generating function of the negative binomial distribution with parameters k/m and (1 − e^− λmt)^− 1.

Relaxing Irwin’s implicit assumption that all individuals were exposed to the same accident risk, Xekalaki ([1981]) treated the parameter λ as referring to a variable risk exposure according to an exponential distribution with density ae^− aλ, a > 0 and obtained the generalized Waring distribution as the accident distribution. In particular,

$$\begin{array}{l}{G}_{X_t}\left(s\right)=a{\displaystyle \underset{0}{\overset{\infty }{\int }}{e}^{-\mathit{a\lambda }}{\left[e^{\mathit{\lambda mt}}-s\left(e^{\mathit{\lambda mt}}-1\right)\right]}^{-k/m}\mathit{d\lambda }}\\ =\frac{a{\left(1-s\right)}^{-k/m}}{\mathit{mt}}{\displaystyle \underset{0}{\overset{\infty }{\int }}{e}^{-\lambda \frac{a+\mathit{kt}}{\mathit{mt}}}{\left(1-\frac{s}{s-1}{e}^{-\lambda }\right)}^{-k/m}\mathit{d\lambda }}\\ =\frac{a{\left(1-s\right)}^{-k/m}}{\mathit{mt}}\frac{\Gamma \left(\left(a+\mathit{kt}\right)/\left(\mathit{mt}\right)\right)}{\Gamma \left(1+\left(a+\mathit{kt}\right)/\left(\mathit{mt}\right)\right)}\mathit{F}_1^2\left(\frac{k}{m},\frac{a+\mathit{kt}}{\mathit{mt}};\frac{a+\mathit{kt}}{\mathit{mt}}+1;\frac{s}{s-1}\right)\\ =\frac{a}{a+\mathit{mt}}\mathit{F}_1^2\left(\frac{k}{m},1;\frac{a}{\mathit{mt}}+\frac{k}{m}+1;s\right)\end{array}$$

which is the probability generating function of the $\mathit{UGWD}\left(\frac{k}{m},1;\frac{a}{\mathit{mt}}\right)$.

This model was considered by Panaretos ([1989]) for the description of the evolution of surnames. Faddy ([1997]) provided a unifying approach to under- and over-dispersion relative to the Poisson distribution within a scheme of a similar nature, which generalizes the simple Poisson process that underpins the Poisson distribution. He demonstrated that any count distribution can be obtained by a suitable choice of f _λ (x, t) and provided an expression for the system of Kolmogorov forward differential equations in terms of a matrix-exponential function.

Finally, Winkelmann ([1995]) looked at under- and over-dispersion using renewal theory by exploring the link between duration dependence and dispersion. He demonstrated that discrepancies between observed and nominal variances are conveyed by a hazard function of the waiting times that is not constant, but instead is a decreasing function of time inducing over-dispersion or an increasing function of time inducing under-dispersion.

6.3 The “Spells” model

Further, Xekalaki ([1983a]) considered a variant of the “spells” model due to Cresswell and Froggatt ([1963]) that rejects the presence of proneness and contagion.

Assume that every individual is liable to spells and that the number of spells in a given time period (0, t) is a Poisson variable with parameter θt, θ > 0. Suppose that no accidents occur outside spells and that the probability of an accident within a spell depends on the risk exposure of the particular individual. In particular, suppose that within a spell a person can have

$$\mathit{or}\left.\begin{array}{l}0\mathrm{accidents}\mathrm{with}\mathrm{probability}1-mlog\left(1+\lambda \right)\\ n\mathrm{accidents}\left(n\ge 1\right)\mathrm{with}\mathrm{probability}m{\left\{\lambda /\left(1+\lambda \right)\right\}}^n/n\end{array}\right\}\text{,}$$

0 < m < 1/log(1 + λ), λ > 0, where λ is the external risk parameter for the given individual. Assume further that the numbers of accidents arising out of different spells are independent and independent of the number of spells. Then, if differences in the risk exposure can be described by a beta distribution of the second kind with probability density function, {Γ(a + ρ)v^a − 1(1 + ν)^{− (a + ρ)}}/{Γ(ρ)Γ(a)}, a, ρ > 0, the resulting accident distribution will have probability generating function given by

$$\left\{{\rho }_{\left(a\right)}\mathit{F}_1^2\left(a,\mathit{\theta mt};a+\mathit{\theta mt}+\rho ;s\right)\right\}/{\left(\rho +\mathit{\theta mt}\right)}_{\left(a\right)}\text{.}$$

Hence, in a unit time period, the number of accidents follows the UGWD(a, θm; ρ).

It is worth noticing that the form of the distribution of λ in the last two models is more general than that considered by the proneness model. It is however, a reasonable choice as it implies a beta distribution of the first kind (Pearson Type I) for the parameter q = λ/(1 + λ) of the negative binomial distribution of X|λ.

6.4 Deciding about the underlying model

It is evident from the above, that three completely different sets of hypotheses give rise to exactly the same form of distribution and that while the UGWD may be a plausible model if accident proneness is a accepted as an established fact, a satisfactory fit of it is not to be taken as evidence for the validity of the proneness hypothesis. How can we then discriminate?

Statisticians have always been excited to look for ways of discriminating among different models that give rise to the same distribution. Most attempts seem to have been concentrated on distinguishing between the proneness and contagion models generating the negative binomial distribution. The papers by Bates and Neyman ([1952]) and Bates ([1955]) cover part of the work that has been done on the subject, though they primarily focus on distinguishing between different forms of contagion. Shaw and Sichel’s ([1971]) attempt was on proving or disproving proneness by ranking individual accident performance on a scale based on their average interval between successive accidents. However, the first systematic study on how one can discriminate between the proneness and contagion models of the negative binomial distribution appears to be that by Cane ([1974]).

She demonstrated, however, that one cannot distinguish between the two models, even with knowledge of the time sequence of accidents. She demonstrated, in particular, that the conditional distribution of the times, t _i , i = 1, 2, …, n at which accidents occurred in a time period (0, T) is the same in both cases, namely that of an ordered sample from a uniform distribution over (0, T) with probability density function n ! T^− n. In fact, this is the case for any compound Poisson accident distribution whose compounding distribution has finite moments (Cane [1977]), hence also for the UGWD(a, k; ρ).

This implies that the availability of information on the times of the occurrence of accidents is not sufficient to guide one’s choice between the proneness and contagion models.

However, as demonstrated by Xekalaki ([1983a]), there appears to exist a possibility in the framework of the Spells model. Consider, in particular, the problem of finding the joint distribution of times t _i , i = 1, 2, …, n of accidents by individuals with n accidents in a unit period of time under the spells model. For fixed λ, accidents occur as events in a generalized Poisson process: $X\left(t\right)={\displaystyle \sum _{i=1}^{N\left(t\right)}{Y}_i},N\left(t\right)\sim \mathit{Poisson}\left(\mathit{\theta t}\right)$, where θ > 0, t ≥ 0 and Y _i are identically and independently distributed with probability density function given by {Γ(a + ρ)v^a − 1(1 + ν)^{− (a + ρ)}}/{Γ(ρ)Γ(a)}, a, ρ > 0. Consequently, the required probability function can be written as ${\displaystyle {\int }_0^{\infty }{\left(1+\lambda \right)}^{-\mathit{\theta m}\left(1-t_n\right)}\left[{\displaystyle \prod _{i=1}^n\left\{\mathit{\lambda m\theta }{\left(1+\lambda \right)}^{-\mathit{\theta m}\left(t_i-t_{i-1}\right)-1}d{t}_i\right\}}\right]\mathit{dH}\left(\lambda \right)}$, with H(⋅) denoting the distribution function of the beta distribution of the second kind defined as above. Hence, the required probability is $\left\{\frac{{\left(\mathit{\theta m}\right)}^n{\rho }_{\left(a\right)}{a}_{\left(n\right)}}{{\left(\mathit{\theta m}+\rho \right)}_{\left(a+n\right)}}\right\}d{t}_1\dots d{t}_n$. Therefore, conditional on n accidents during a time period from 0 to 1, the joint pdf of t _i , i = 1, 2, …, n, is n ! (θm)ⁿ/(θm)_(n).

The obtained form differs from that arising under the proneness and contagion models. This fact is itself is very interesting as far as establishing the presence of spells is concerned, as it implies the following: if an observed accident distribution of the UGWD type has arisen from the spells model, the time intervals (0, t _i ), i = 1, 2, …, n, given a total of n accidents, will be jointly distributed with the above density function. Any departure from this distribution is, then, evidence against the spells model. Of course, if on the available evidence one has to reject this form in favor of that obtained by Cane, then one is faced again with the question: “proneness or contagion?” This cannot be answered by studying the distribution of t _i .

6.5 What does Irwin’s accident model offer beyond a good fit to the data?

The innovation brought by Irwin’s accident proneness model does not merely lie in the better fit it provides to accident data, but in the possibility of partitioning the total variance (σ²) into three additive components due to proneness $\left({\sigma }_{\nu }^2\right)$, liability $\left({\sigma }_{\lambda }^2\right)$ and randomness $\left({\sigma }_R^2\right)$ thus,

$${\sigma }^2={\sigma }_{\lambda }^2+k^2{\sigma }_{\nu }^2+{\sigma }_R^2\text{,}$$

Where

$$\begin{array}{l}{\sigma }_{\lambda }^2=\mathit{ak}\left(a+1\right){\left(\rho -1\right)}^{-1}{\left(\rho -2\right)}^{-1}\\ {\sigma }_{\nu }^2=a{\left(a+\rho -1\right)}^{-1}{\left(\rho -1\right)}^{-2}{\left(\rho -2\right)}^{-1}\\ {\sigma }_R^2=\mathit{ak}{\left(\rho -1\right)}^{-1}\\ {\sigma }^2=\mathit{ak}\left(a+\rho -1\right)\left(k+\rho -1\right){\left(\rho -1\right)}^{-2}{\left(\rho -2\right)}^{-1}\text{.}\end{array}$$

There is still, however, a problem due to the fact that the UGWD(a, k; ρ) is symmetrical in a and k (UGWD(a, k; ρ) ∼ UGWD(k, a; ρ)). Hence, although one may consider that ${\sigma }_{\lambda }^2+k^2{\sigma }_{\nu }^2$ represents the variance component due to all non-random factors, the mathematics alone cannot determine whether ${\sigma }_{\lambda }^2$ represents the liability component and $k^2{\sigma }_{\nu }^2$ the proneness component or vice versa. As a consequence, distinguishable estimates for the non-random variance components ${\sigma }_{\lambda }^2$ and ${\sigma }_{\nu }^2$ cannot be obtained unless subjective judgement is made. This problem was addressed by Xekalaki ([1984a]) with the introduction of her bivariate form of the generalized Waring distribution.

7. The bivariate generalized Waring distribution

Generalizing further Irwin’s ([1963]) generalization of Waring’s expansion, we have for k, m, a > 0,

$$\begin{array}{l}\frac{1}{{\left(x-a\right)}_{\left(k+m\right)}}={\displaystyle \sum _{\ell =0}^{\infty }{\displaystyle \sum _{r=0}^{\ell }\frac{a_{\left(\ell \right)}{\left(-1\right)}^{\ell }}{\ell !}}}\left(\begin{array}{c}\hfill \ell \hfill \\ \hfill r\hfill \end{array}\right){\mathrm{\Delta }}^r\frac{1}{x_{\left(k\right)}}{\mathrm{\Delta }}^{\ell -r}\frac{1}{{\left(x+k+r\right)}_{\left(m\right)}}\\ ={\displaystyle \sum _{r=0}^{\infty }{\displaystyle \sum _{\ell =0}^{\infty }\frac{a_{\left(r+\ell \right)}{\left(-1\right)}^{r+\ell }}{r!\ell !}}}{\mathrm{\Delta }}^r\frac{1}{x_{\left(k\right)}}{\mathrm{\Delta }}^{\ell }\frac{1}{{\left(x+k+r\right)}_{\left(m\right)}}\\ ={\displaystyle \sum _{r=0}^{\infty }{\displaystyle \sum _{\ell =0}^{\infty }\frac{a_{\left(r+\ell \right)}{k}_{\left(r\right)}{m}_{\left(\ell \right)}}{x_{\left(k+m+r+\ell \right)}}}}\frac{1}{r!}\frac{1}{\ell !}\end{array}$$

If x > a, the above series is convergent. Then, by letting ρ = x − a > 0 and multiplying both sides by ρ_(k + m), leads to a double series of positive terms converging to unity. The general term of the series therefore can be regarded as defining a bivariate discrete probability distribution with probability function

$$p_{r,\ell }=\frac{{\rho }_{\left(k+m\right)}}{{\left(a+\rho \right)}_{\left(k+m\right)}}\frac{a_{\left(r+\ell \right)}{k}_{\left(r\right)}{m}_{\left(\ell \right)}}{{\left(a+k+m+\rho \right)}_{\left(r+\ell \right)}}\frac{1}{r!}\frac{1}{\ell !},a,k,m,\rho >0,r,\ell =0,1,2,\dots$$

In the remainder of the paper, we refer to this distribution as the bivariate generalized Waring distribution with parameters a, k, m and ρ and we denote it by BGWD(a; k, m; ρ).

7.1 The BGWD in relation to accident theory

Assume that individuals of proneness ν and liability λ _i |ν for a period i of observation incur, over two non-overlapping time periods, accidents X, Y according to a double Poisson distribution $G_{\left(X,Y\right)|{\lambda }_1,{\lambda }_2,\nu }\left(s,t\right)=exp\left\{\left({\lambda }_1|\nu \right)\left(s-1\right)+\left({\lambda }_2|\nu \right)\left(t-1\right)\right\},{\lambda }_1,{\lambda }_2>0$. Assume further that the liability parameters λ₁|ν, λ₂|ν are independently gamma distributed with densities ${\left(\Gamma \left({\theta }_i\right){\nu }^{{\theta }_i}\right)}^{-1}{e}^{-{\lambda }_i|\nu }{\lambda }_i^{{\theta }_i-1},{\theta }_1\equiv k,{\theta }_2\equiv m,\nu >0$, whence for individuals with the same proneness ν, but varying liabilities, the numbers of occurring accidents over the two periods are jointly distributed as the double negative binomial with probability generating function

$$G_{\left(X,Y\right)|\nu }\left(s,t\right)={\left\{1+\nu \left(1-s\right)\right\}}^{-k}{\left\{1+\nu \left(1-t\right)\right\}}^{-m}\text{.}$$

Letting now the proneness parameter ν be beta distributed with density function {Γ(a + ρ)v^a − 1(1 + ν)^{− (a + ρ)}}/{Γ(ρ)Γ(a)}, a, ρ > 0, the probability generating function of the joint distribution of accidents over the two periods takes the form

$$\begin{array}{l}{G}_{\left(X,Y\right)}\left(s,t\right)=\frac{\Gamma \left(\rho +a\right)}{\Gamma \left(\rho \right)\Gamma \left(a\right)}{\displaystyle {\int }_0^{+\infty }{\nu }^{a-1}{\left(1+\nu \right)}^{-\left(a+\rho \right)}{\left\{1+\nu \left(1-s\right)\right\}}^{-k}{\left\{1+\nu \left(1-t\right)\right\}}^{-m}\mathit{d\nu }}\\ =\frac{{\rho }_{\left(k+m\right)}}{{\left(a+\rho \right)}_{\left(k+m\right)}}{F}_1\left(a;k,m;a+k+m+\rho ;s,t\right)~\mathit{BGWD}\left(a;k,m;\rho \right)\text{,}\end{array}$$

where $F_1\left(a;b,c;d;u,v\right)={\displaystyle {\sum }_{r,s=0}^{\infty }\left\{a_{\left(r+s\right)}{b}_{\left(r\right)}{c}_{\left(s\right)}{u}^r{v}^s\right\}/\left\{d_{\left(r+s\right)}r!s!\right\}}$ is Appell’s hypergeometric series and h(l) = Γ(h + l)/Γ(h), h > 0, l ∈ R.

Regarding separate estimation of the contribution of proneness, liability and randomness in a given accident situation over a period of observation whenever proneness is accepted as an established fact, Xekalaki ([1984a]) showed that rearranging the observed distribution in two non-overlapping sub-intervals and fitting the BGWD(a; k, m; ρ) to the resulting bivariate accident distribution does enable separate estimation of the variance components. This is demonstrated in Table 1.

Table 1 Estimators of the components of the variance of the generalized waring distribution

Further models leading to the BGWD provided by Xekalaki ([1984c]), provide the framework within which one can also obtain the BGWD as an accident distribution under the contagion and the spells accident theories.

8. The multivariate generalized Waring distribution

The n-variate version of the genaralized Waring distribution introduced and studied by Xekalaki ([1986]) is also obtained as an inverse factorial distribution. Its probability generating function is given by

$$G\left(\underset{¯}{t}\right)=\frac{{\rho }_{\left({\displaystyle \sum k_i}\right)}}{{\left(a+\rho \right)}_{\left({\displaystyle \sum k_i}\right)}}{F}_D\left(a;k_1,\dots ,k_n;a+{\displaystyle \sum _{i=1}^n{k}_i}+\rho ;\underset{¯}{t}\right)$$

with %$F_D\left(a;{\beta }_1,\dots ,{\beta }_n;\gamma ;\underset{¯}{t}\right)$ denoting Lauricella’s hypergeometric function given by

$$F_D\left(a;{\beta }_1,\dots ,{\beta }_n;\gamma ;\underset{¯}{t}\right)={\displaystyle \sum _{r_1,\dots ,r_n}\frac{a_{\left({\displaystyle \sum r_i}\right)}}{{\gamma }_{\left({\displaystyle \sum r_i}\right)}}{\displaystyle \prod _{i=1}^n\frac{{\left({\beta }_i\right)}_{\left(r_i\right)}{t}_i^{r_i}}{\left(r_i\right)!}}}$$

The probability function of it is given by

$$\begin{array}{l}{P}_{\underset{¯}{r}}\equiv P\left(\underset{¯}{X}=\underset{¯}{r}\right)=\frac{{\rho }_{\left({\displaystyle \sum k_i}\right)}}{{\left(a+\rho \right)}_{\left({\displaystyle \sum k_i}\right)}}\frac{a_{\left({\displaystyle \sum r_i}\right)}{\left(k_1\right)}_{\left(r_i\right)}\dots {\left(k_n\right)}_{\left(r_n\right)}}{{\left(a+\rho +{\displaystyle \sum k_i}\right)}_{\left({\displaystyle \sum r_i}\right)}\left(r_1\right)!\dots \left(r_n\right)!},r_i=0,1,2,\dots ;i=1,\dots ,n\end{array}$$

and its probabilities are related by the following first order recurrences, which facilitate their computation

$$\frac{P_{l_1,l_2,\dots ,l_{h-1},l_h+1,l_{h+1},\dots ,l_n}}{P_{l_1,l_2,\dots ,l_n}}=\frac{\left[a+{\displaystyle \sum _{i=1}^n{l}_i}\right]\left(k_n+l_n\right)}{\left[a+{\displaystyle \sum _{i=1}^n{k}_i}+{\displaystyle \sum _{i=1}^n{l}_i}\right]\left(l_n+1\right)},l=0,1,2,\dots ;i=1,2,\dots ,n$$

An interesting aspect of the bivariate and multivariate versions of the generalized Waring distribution is that their marginal distributions (conditional and unconditional) as well as their convolution are of the same form (UGWD’s), properties that exhibit a symmetry analogous to that existing in the case of the multivariate normal distribution. Further, the generalized Waring distribution is self-decomposable (Xekalaki [1983b]).

9. The Generalized Waring Process (gWp)

Looking into how temporally evolving data from the wide spectrum of application contexts that can reasonably be viewed from the perspective of the frameworks discussed in Sections 6, 7 and 8 can be treated, Xekalaki and Zografi ([2008]) defined and studied the generalized Waring process. In establishing its definition, the structural properties of both the bivariate and the multivariate versions of the generalized Waring distribution played a significant role. This process, analogously to the case of Poisson and Pólya processes, which can be obtained as limiting cases of it, was shown to be a Markov process.

Let {N(t), t ≥ 0} be a counting process. This is said to be a generalized Waring process with parameters a, k, ρ > 0, denoted by gWp(a, k; ρ), if (i) N(0) = 0, (ii) N(t) is a Markov process, and (iii) N(t + h) − N(t) has the generalized Waring distribution with parameters a, k; ρ for h > 0, t ≥ 0. The process starts at 0, it has stationary increments and

$$P\left(N\left(t\right)=n\right)=\frac{{\rho }_{\left(\mathit{kt}\right)}}{{\left(\rho +a\right)}_{\left(\mathit{kt}\right)}}\frac{a_{\left(n\right)}{\left(\mathit{kt}\right)}_{\left(n\right)}}{{\left(a+\rho +\mathit{kt}\right)}_{\left(n\right)}}\frac{1}{n!}$$

i.e., N(t) has a generalized Waring distribution with parameters a, kt; ρ.

The transition probabilities of the generalized Waring process are given by

$$\begin{array}{l}{p}_{m,n}\left(s,s+t\right)=P\left\{N\left(s+t\right)=n|N\left(s\right)=m\right\}=\frac{\Gamma \left(a+n\right)}{\Gamma \left(a+m\right)}\frac{{\left(\mathit{kt}\right)}_{\left(n-m\right)}}{\left(n-m\right)!}\frac{{\left(\rho +\mathit{ks}\right)}_{\left(a+m\right)}}{{\left(\rho +\mathit{ks}+\mathit{kt}\right)}_{\left(a+n\right)}}\\ p_{0,n}\left(0,t\right)=P\left\{N\left(t\right)=n|N\left(0\right)=0\right\}=\frac{{\rho }_{\left(\mathit{kt}\right)}}{{\left(\rho +a\right)}_{\left(\mathit{kt}\right)}}\frac{a_{\left(n\right)}{\left(\mathit{kt}\right)}_{\left(n\right)}}{{\left(a+\rho +\mathit{kt}\right)}_{\left(n\right)}}\frac{1}{n!}=P\left(N\left(t\right)=n\right)\end{array}$$

with the last equality indicating that the generalized Waring process is a non-homogenous Markov process. Its mean and variance are respectively

$$E\left[N\left(t\right)\right]=\frac{\mathit{akt}}{\rho -1}\mathrm{and}\mathit{Var}\left[N\left(t\right)\right]=\frac{\mathit{akt}\left(\rho +\mathit{kt}-1\right)\left(\rho +\alpha -1\right)}{{\left(\rho -1\right)}^2\left(\rho -2\right)}$$

Note that since the generalized Waring process is a stationary process and its mean is of the form E[N(t)] = ηt, the above formula implies that its intensity is η = ak/(ρ − 1). Its variance can be split into three additive components, thus

$$\mathit{Var}\left[N\left(t\right)\right]={\sigma }_{\Lambda \left(t\right)}^2+{\left(\mathit{kt}\right)}^2{\sigma }_{\nu }^2+{\sigma }_R^2$$

with the liability and random components dependent on time. In particular,

$${\sigma }_{\Lambda \left(t\right)}^2=\mathit{akt}\left(a+1\right){\left(\rho -1\right)}^{-1}{\left(\rho -2\right)}^{-1};{\sigma }_{\nu }^2=a\left(a+\rho -1\right){\left(\rho -1\right)}^{-2}{\left(\rho -2\right)}^{-1};{\sigma }_R^2=\mathit{akt}{\left(\rho -1\right)}^{-1}\text{.}$$

9.1 The generalized Waring process in an accident proneness context

We consider a population which is inhomogeneous with respect to personal and environmental attributes affecting the occurrence of accidents. The terms “accident proneness” and “accident liability” are again used to refer respectively to a person’s predisposition to accidents, and to a person’s exposure to external risk of accident with the conditional distribution of the random variable λ given ν describing differences in external risk factors among individuals. Liability fluctuations over a time interval (t, t + h) depend on the length h of the interval and are described by a distribution for λ|ν with probability density function λ^kh − 1e^{− λ/(νh)}(νh)^− kh/Γ(kh). Allowing further the parameter ν have a beta distribution of the second kind with parameters a and ρ and density function ϕ given by

ϕ(ν) = Γ(a + ρ)ν^a − 1(1 + ν)^{− (a + ρ)}/[Γ(a)Γ(ρ)], a, ρ ≥ 0, we obtain for the distribution of the number of accidents N(t):

$$P\left(N\left(t+h\right)-N\left(t\right)=n\right)=\frac{{\rho }_{\left(\mathit{kh}\right)}}{{\left(a+\rho \right)}_{\left(\mathit{kh}\right)}}\frac{a_{\left(n\right)}{\left(\mathit{kh}\right)}_{\left(n\right)}}{{\left(a+\rho +\mathit{kh}\right)}_{\left(n\right)}}\frac{1}{n!}$$

and

$$P\left(N\left(t\right)=n\right)=P_n\left(t\right)=\frac{{\rho }_{\left(\mathit{kt}\right)}}{{\left(a+\rho \right)}_{\left(\mathit{kt}\right)}}\frac{a_{\left(n\right)}{\left(\mathit{kt}\right)}_{\left(n\right)}}{{\left(a+\rho +\mathit{kt}\right)}_{\left(n\right)}}\frac{1}{n!},n=0,1,\dots$$

So, the process arising in the context of this model, satisfies the defining conditions of the generalized Waring process.

9.2 The generalized Waring process in the context of a spells model

Xekalaki and Zografi ([2008]) showed that the generalized Waring process could also be used in modeling temporally evolving data in the context of a spells model. Assume again that each person is liable to spells and that no accidents can occur outside spells. Let S(t), t = 0, 1, 2, …, the number of spells up to a given moment t, be a homogeneous Poisson process with rate k/m, k > 0, the number X _i of accidents within a spell i be a random variable with a logarithmic series distribution with parameters m and ν and probability function given by %$P\left(X_i=n\right)=\frac{m}{n}{\left(\frac{\nu }{1+\nu }\right)}^n,n\ge 1$ with P(X _i  = 0) = 1 − m log(1 + ν), ν > 0, 0 < m < 1/log(1 + ν), and the numbers of accidents arising out of different spells be independent and independent of the number S(t) of spells. Here ν is regarded as the external risk parameter, too, which they assumed varying according to a beta distribution of the second kind with parameters a and ρ and probability density function given by Γ(a + ρ)ν^a − 1(1 + ν)^{− (a + ρ)}/[Γ(a)Γ(ρ)], a, ρ ≥ 0. They then showed that the above framework leads to a process conforming with the postulates of the generalized Waring process, thus demonstrating its potential application in the context of the Spells model.

10. An application: modeling the counting process {N(s), s > 0} associated with the access pattern of a web site

As an illustration of the application potential of the generalized Waring process in other fields by appropriately adjusting the concepts and terminology used in this paper so as to have natural interpretations, we outline an example of a model for temporally evolving data on web access patterns provided by Xekalaki and Zografi ([2008]).

In this context, {N(s), s > 0} is the counting process associated with the access pattern of a web site, where, for any t > 0, N(t) represents the number of visits that the web pages on this particular site get within the interval (0, t). Note that the generalized Waring distribution was cited in Ajiferuke et al. ([2004]) as used by them to fit observed website visitation data for a given period, i.e, to model counts N(t₀) of web visits on a given fixed time interval (0, t₀).

Except for chance, visits to a web site can be regarded as affected by the intrinsic appeal of the particular site to web users (corresponding to proneness) as well as by exogenous factors (corresponding to external factors) such as, links provided by other sites to the particular site, how well the site is advertised etc.

Letting ν denote the intrinsic factors and λ|ν the exogenous factors. Then assuming that N(t)|λ follows a Poisson(λ(t)) distribution, where λ(t) = λt with λ|ν following a gamma distribution with density λ^kt − 1e^{− λ/(νt)}(νt)^− kt/Γ(kt), and with ν following a beta distribution of the second kind with density Γ(a + ρ)ν^a − 1(1 + ν)^{− (a + ρ)}/[Γ(a)Γ(ρ)], a, ρ ≥ 0, then the unconditional distribution of N(t) is the GWD(a, kt; ρ), i.e. the process {N(t), t ≥ 0} is a generalized Waring process.

10.1 The data

The log files representing the hits on an e-shop site for the period from March 31, 2006 to April 30, 2006 have been used to fit this model. (A log file typically contains information on the times of visits per IP address per day). On the basis of such log files, the visits per day made by each of 468 IP addresses to a web site during the above period were enumerated yielding 468 paths of visits N _i (t _j ) made by IP address i up to and including time t _j denoted by {N _i (t _j ), i = 1, 2, …, 468; j = 1, 2, …, 31}.

Moment estimates of the parameters of the generalized Waring process were obtained employing an estimation procedure for spatial point process data termed in the literature as the centered reduced moment method. The method introduced and studied by Ripley ([1976], [1977]) utilizes the intensity of the process and the mean number of further points within distance s of an arbitrary point of the process. In particular, the method utilizes the moment estimators %$E\left(\widehat{N}\left(s\right)\right)={\widehat{\mu }}_1=\widehat{\eta }s=\mathit{ns}/h,E\left({\widehat{N}}^2\left(s\right)\right)={\widehat{\mu }}_2=X/n_{\left(2\right)},E\left({\widehat{N}}^3\left(s\right)\right)={\widehat{\mu }}_3=\left(Z-X\right)/n_{\left(3\right)}$ with %$X={\displaystyle \sum _{i=1}^n{\displaystyle \sum _{i\ne j}{\varphi }_s^2\left(x_i,x_j\right)}},Z={\displaystyle \sum _{i=1}^n\left({\displaystyle \sum _{j\ne i}{\varphi }_s\left(x_i,x_j\right)}\right)}\left({\displaystyle \sum _{k\ne i}{\varphi }_s\left(x_i,x_k\right)}\right)$, where the quantities involved in the above equations represent weights defined, for each value x _i in the collection of points {x _i  : i = 1, 2, …, n} of the process within a time interval of length h, as follows: For each x _i in {x _i  : i = 1, 2, …, n} and a given s > 0, consider the interval of center x _i and length s and assign to every point x _j , j ≠ i in this interval the weight ϕ _s (x _i , x _j ) = ω(x _i , x _j )^− 1, where ω(x _i , x _j ) is the number of other points {x _k , k ≠ i, k ≠ j} of the process that are included in the interval of length |x _i  − x _j | and center x _i (see also Diggle and Chetwynd [1991]; Chetwynd and Diggle [1998], among others). The standard errors of the thus obtained parameter estimators can in principle be determined by simulation, but the associated computations are formidable. Approximation formulas exist only for the case of homogeneous planar Poisson process, while, for the class of stationary Cox process, there is no obvious way to obtain estimable expressions as noted by Chetwynd and Diggle ([1998]).

The observed paths were compared to the corresponding time series of simulated realizations of the generalized Waring process over the same time segment. For each IP address, 100 simulated realizations of the gWp(a, k; ρ), were obtained and each of the observed time series paths was compared to the corresponding simulated ones. On average, the realizations of the generalized Waring process exhibited similar structural characteristics, notably recognizable, to those of the paths of the observed time series. For illustration purposes, the path of the observed time series associated with one of the IP addresses considered is presented in Figure 1. In the graph, the path is superimposed by a sample of three of the 100 corresponding simulated realizations of the gWp(a, k; ρ). Inspection of the graph provides a visual appreciation of the degree of similarity in the structural characteristics of the path of the observed and the realized time series.

Figure 1

Observed and simulated paths of the gWp(3.87, 0.83; 4.21) corresponding to the selected IP address (Xekalaki and Zografi[2008]).

Following Lewis ([1972]), Brillinger ([1978]) and Andersen et al. ([1993]), the closeness of the observed and realized time series was also checked using diagnostic plots based on the inverse-intensity residuals computed for each value x _j in the collection of points {x _j  : j = 1, 2, …, n} of the process given by %$R_{\widehat{\theta }}\left(B_j,{\eta }^{-1}\right)={\displaystyle \sum _{x_i\in B_j}\widehat{\eta }\left(x_i\right)}-{\displaystyle {\int }_{B_j}^j{I}_{R^{+}}\left(\widehat{\eta }\left(x\right)\right)\mathit{dx}}$ where %$B_j=\left(0,x_j\right),\widehat{\theta }={\left(\widehat{a},\widehat{k},\widehat{\rho }\right)}^{-1},\widehat{\eta }\left(x\right)=\eta \left(x,\widehat{\theta }\right)$ is the fitted intensity and %$I_{R^{+}}\left(\cdot \right)$ is the indicator function. These plots exhibit similar results. The plot corresponding to the data associated to the IP address considered is shown in Figure 2.

Figure 2

Plot of inverse-intensity residuals corresponding to the selected IP address (Xekalaki and Zografi[2008]).