Abstract
Within the framework of probability models for overdispersed count data, we propose the generalized fractional Poisson distribution (gfPd), which is a natural generalization of the fractional Poisson distribution (fPd), and the standard Poisson distribution. We derive some properties of gfPd and more specifically we study moments, limiting behavior and other features of fPd. The skewness suggests that fPd can be left-skewed, right-skewed or symmetric; this makes the model flexible and appealing in practice. We apply the model to real big count data and estimate the model parameters using maximum likelihood. Then, we turn to the very general class of weighted Poisson distributions (WPD’s) to allow both overdispersion and underdispersion. Similarly to Kemp’s generalized hypergeometric probability distribution, which is based on hypergeometric functions, we analyze a class of WPD’s related to a generalization of Mittag–Leffler functions. The proposed class of distributions includes the well-known COM-Poisson and the hyper-Poisson models. We characterize conditions on the parameters allowing for overdispersion and underdispersion, and analyze two special cases of interest which have not yet appeared in the literature.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
1 Introduction and mathematical background
The negative binomial distribution is one of the most widely used discrete probability models that allow departure from the mean-equal-variance Poisson model. More specifically, the negative binomial distribution models overdispersion of data relative to the Poisson distribution. For clarity, we refer to the extended negative binomial distribution with probability mass function
where \(r>0\). If \(r \in \{1,2,\dots \}\), x is the number of failures which occur in a sequence of independent Bernoulli trials to obtain r successes, and p is the success probability of each trial.
One limitation of the negative binomial distribution in fitting overdispersed count data is that the skewness and kurtosis are always positive. An example is given in Sect. 2.1.1, in which we introduce two real world data sets that do not fit a negative binomial model. The data sets reflect reported incidents of crime that occurred in the city of Chicago from January 1, 2001 to May 21, 2018. These data sets are overdispersed but the skewness coefficients are estimated to be respectively -0.758 and -0.996. Undoubtedly, the negative binomial model is expected to underperform in these types of count populations. These data sets are just two examples in a yet to be discovered non-negative binomial world, thus demonstrating the real need for a more flexible alternative for overdispersed count data. The literature on alternative probabilistic models for overdispersed count data is vast. A history of the overdispersed data problem and related literature can be found in Shmueli et al. (2005). In this paper we consider the fractional Poisson distribution (fPd) as an alternative. The fPd arises naturally from the widely studied fractional Poisson process (Saichev and Zaslavsky 1997; Repin and Saichev 2000; Jumarie 2012; Laskin 2003; Beghin and Orsingher 2009; Cahoy et al. 2010; Meerschaert et al. 2011). It has not yet been studied in depth and has not been applied to model real count data. We show that the fPd allows big (large mean), both left- and right-skewed overdispersed count data making it attractive for practical settings, especially now that data are becoming more available and bigger than before. fPd’s usually involve one parameter; generalizations to two parameters are proposed in Beghin and Orsingher (2009) and Herrmann (2016). Here, we take a step forward and further generalize the fPd to a three parameter model, proving the resulting distribution is still overdispersed.
One of the most popular measures to detect the departures from the Poisson distribution is the so-called Fisher index which is the ratio of the variance to the mean \((\lessgtr 1)\) of the count distribution. As shown in the crime example of Sect. 2.1.1, the computation of the Fisher index is not sufficient to determine a first fitting assessment of the model, which indeed should take into account at least the presence of negative/positive skewness. To compute all these measures, the first three factorial moments should be considered. Consider a discrete random variable X with probability generating function (pgf)
where \(\{a_k\}\) is a sequence of real numbers such that \(a_0=1.\) Observe that \(Q(t)=G_X(1+t)\) is the factorial moment generating function of X. The k-th moment is
where S(k, r) are the Stirling numbers of the second kind (Di Nardo and Senato 2006). By means of the factorial moments it is straightforward to characterize overdispersion or underdispersion as follows: letting \(a_2 > a_1^2\) yields overdispersion whereas \(a_2 < a_1^2\) gives underdispersion. Let \(c_2\) and \(c_3\) be the second and third cumulant of X, respectively. Then, the skewness can be expressed as
If the condition
is fulfilled, the probability mass function of X can be written in terms of its factorial moments (Daley and Narayan 1980):
As an example, the very well-known generalized Poisson distribution which accounts for both under and overdispersion (Maceda 1948; Consul and Jain 1973), put in the above form has factorial moments given by \(a_0=1\) and
where \(h(\lambda _2) = \infty \) and \(k =1,2,\ldots \), if \(\lambda _2>0\). While \(h(\lambda _2) = M-k\) and \(k=1, \ldots , M\), if \(\max (-1, -\lambda _1/M) \le \lambda _2 < 0\) and M is the largest positive integer for which \(\lambda _1+M\lambda _2 > 0\).
Another example is given by the Kemp family of generalized hypergeometric factorial moments distributions (GHFD) (Kemp and Kemp 1974) for which the factorial moments are given by
where \( \Gamma \left[ (a); (b) \right] =\prod _{i=1}^p \Gamma (a_i)/\prod _{j=1}^q \Gamma (b_j)\), with \(a_1, \ldots , a_p, b_1, \ldots , b_q \in {{\mathbb {R}}}\) and p, q non negative integers. The factorial moment generating function is \(Q(t) = \, _pF_q \left[ (a) ;(b); \lambda t \right] \), where
and \((a)_m=a(a+1)\cdots (a+m-1), m \ge 1.\) Both overdispersion and underdispersion are possible, depending on the values of the parameters (Tripathi and Gurland 1979). The generalized fractional Poisson distribution (gfPd), which we introduce in the next section, lies in the same class of the Kemp’s GHFD but with the hypergeometric function in (9) substituted by a generalized Mittag–Leffler function (also known as three-parameter Mittag–Leffler function or Prabhakar function). In this case, as we have anticipated above, the model is capable of not only describing overdispersion but also having a degree of flexibility in dealing with skewness.
It is worthy to note that there exists a second family of Kemp’s distributions, still based on hypergeometric functions and still allowing both underdispersion and overdispersion. This is known the Kemp’s generalized hypergeometric probability distribution (GHPD) (Kemp 1968) and it is actually a special case of the very general class of weighted Poisson distributions (WPD). Taking into account the above features, we thus analyze the whole class of WPD’s with respect to the possibility of obtaining under and overdispersion. In Theorem 3.2 we first give a general necessary and sufficient condition to have an underdispersed or an overdispersed WPD random variable in the case in which the weight function may depend on the underlying Poisson parameter \(\lambda \). Special cases of WPD’s admitting a small number of parameters have already proven to be of practical interest, such as for instance the well-known COM-Poisson (Conway and Maxwell 1962) or the hyper-Poisson (Bardwell and Crow 1964) models. Here we present a novel WPD family related to a generalization of Mittag–Leffler functions in which the weight function is based on a ratio of gamma functions. The proposed distribution family includes the above-mentioned well-known classical cases. We characterize conditions on the parameters allowing overdispersion and underdispersion and analyze two further special cases of interest which have not yet appeared in the literature. We derive recursions to generate probability mass functions (and thus random numbers) and show how to approximate the mean and the variance.
The paper is organized as follows: in Sect. 2, we introduce the generalized fractional Poisson distribution, discuss some properties and recover the classical fPd as a special case. These models are fit to the two real-world data sets mentioned above. Sect. 3 is devoted to weighted Poisson distributions, their characteristic factorial moments and the related conditions to obtain overdispersion and underdispersion. Furthermore, the novel WPD based on a generalization of Mittag–Leffler functions is introduced and described in Sect. 3.1: we discuss some properties and show how to get exact formulae for factorial moments by using Faà di Bruno’s formula (Stanley 2012). Two special models are then characterized depending on the values of the parameters and compared to classical models. Finally, some illustrative plots end the paper.
2 Generalized fractional Poisson distribution (gfPd)
Definition 2.1
A random variable \(X_{\alpha , \beta }^\delta \; {\mathop {=}\limits ^{d}} \; \)gfPd\((\alpha , \beta , \delta , \mu )\) if
where
\(w \in {\mathbb {C}}; \mathfrak {R}(\eta ),\mathfrak {R}(\nu ),\mathfrak {R}(\tau ) >0\), is the generalized Mittag–Leffler function (Prabhakar 1971) and \((\tau )_j = \Gamma ( \tau + j )/ \Gamma ( \tau ) \) denotes the Pochhammer symbol.
To show non-negativity, notice that
that is, \( E_{\alpha , \beta }^{\delta } (-\mu )\) is completely monotone. From De Oliveira et al. (2011), it is known that \( E_{\alpha , \beta }^{\delta } (-\mu )\) is completely monotone if \(\alpha , \beta \in (0, \;1], \delta \in (0, \; \beta /\alpha ]\) and thus the pmf in (10) is non-negative.
Note that the probability mass function can be determined using the following integral representation (Polito and Tomovski 2016):
where the Wright function \(\phi \) is defined as the convergent sum (Kilbas et al. 2006)
Remark 2.1
The random variable \(X_{\alpha ,\beta }^\delta \) has factorial moments
Hence the pgf is \(G_{X_{\alpha , \beta }^\delta }(u)= \Gamma (\beta ) E_{\alpha , \beta }^{\delta } ( \mu (u-1) )\), \(|u| \le 1\).
By expressing the moments in terms of factorial moments and after some algebra we obtain
Theorem 2.1
\(X_{\alpha , \beta }^\delta \) exhibits overdispersion.
Proof
We have
and
Thus, the distribution is overdispersed for
Observe that the function \(\beta \text {Beta}(\beta ,\alpha )\) is increasing in \(\beta \) for \(\alpha , \beta \in (0,1)\) as
where \(\psi \) is the digamma function. Note that (21) is positive by formula (1.3.3) of Lebedev (1972) as \(\psi \) is increasing on \((0,\infty )\). Thus
and for \(\delta \in (0, \beta /\alpha )\) the bound (20) is always verified. \(\square \)
2.1 Fractional Poisson distribution
This section analyzes the classical fPd, which is a special case of gfPd, and is obtained when \(\beta =\delta =1\). The fPd can model asymmetric (both left-skewed and right-skewed) overdispersed count data for all mean count values (small and large). The fPd has probability mass function (pmf)
where \(\mu >0\), \(\alpha \in [0,1]\).
Notice that if \(\alpha = 1\), the standard Poisson distribution is retrieved, while for \(\alpha =0\) we have \(X_{0} {\mathop {=}\limits ^{d}} \text {Geo} \left( 1/(1+ \mu )\right) \). Indeed,
Furthermore, the probability mass function can be determined using the following integral representation (Beghin and Orsingher 2010):
where the M-Wright function (Mainardi et al. 2010)
is the probability density function of the random variable \(S^{-\alpha }\) with \(S {\mathop {=}\limits ^{d}} \alpha ^+\)-stable supported in \({\mathbb {R}}^+.\) By using (25), the cumulative distribution function turns out to be
Remark 2.2
From (2.1), the random variable \(X_\alpha \) has factorial moments
Hence the probability generating function is \(G_{X_\alpha }(u)= E_{\alpha ,1}^1 \left( \mu \left( u-1\right) \right) \), \(|u| \le 1\).
With respect to the symmetry structure of \(X_\alpha \), from (4) and (28), the skewness of \(X_\alpha \) reads
Moreover,
which correctly vanishes if \(\alpha =1\), like the ordinary Poisson distribution.
2.1.1 Simulation and parameter estimation
The integral representation (25) allows visualization of the probability mass function of \(X_\alpha \) (see Fig. 1). Figure 1 shows the flexibility of the fPd. The probability distribution ranges from zero-inflated right-skewed (\(\alpha \rightarrow 0\)) to left-skewed (\(\alpha \rightarrow 1\)) and symmetric (\(\alpha = 1\)) overdispersed count data. To compute the integral in (25) by means of Monte Carlo techniques, we use the approximation,
where \(Y_j's \; {\mathop {=}\limits ^{iid}} S^{-\alpha }.\) Note that the random variable S can be generated using the following formula (Kanter 1975; Chambers et al. 1976):
where \(U_1\) and \(U_2\) are independently and uniformly distributed in [0, 1]. Thus, fractional Poisson random numbers can be generated using the algorithm below.
Algorithm:
Step 1. Set \(X=0,\) and \(T=0.\)
Step 2. While \(\lbrace T\le 1 \rbrace \)
$$\begin{aligned} T= & {} T+ V^{1/\alpha } \, S \\ X= & {} \text {ifelse}(T \le 1, X+1, X) \end{aligned}$$Step 3. Repeat steps \(1-2, n\) times.
Note that the random variable V follows the exponential distribution with density function \(\mu \exp (-\mu v), v \ge 0.\) Algorithms for generating random variables from the exponential density function are well-known. Hence, the algorithm allows estimation of the kth moment, i.e., \({\varvec{E}} X_\alpha ^{k}.\)
Figure 2 shows the plot of the skewness coefficient (30) as a function of \(\mu \) and \(\alpha .\) Unlike the negative binomial, the fPd can accommodate both left-skewed and right-skewed count data making it more flexible. Thus, the fPd is more flexible than the negative binomial, especially if the number of failures becomes large.
We applied the fractional Poisson model \(\text {fPd}(\alpha , \mu )\) to two data sets, named Data 1 and Data 2, which are about the reported incidents of crime that occurred in the city of Chicago from 2001 to present.Footnote 1 The sample distributions together with their description are shown in Fig. 3.
Furthermore, we compared \(\text {fPd}(\alpha , \mu )\) with the negative binomial \(\text {NegBinom}(\textit{size},\) \(\textit{mean})\) using the usual chi-square goodness-of-fit test statistic and the maximum likelihood estimates for both models. Note that the chi-square test statistic follows, approximately, a chi-square distribution with \((k - 1-p)\) degrees of freedom where k is the number of cells and p is the number of parameters to be estimated plus one.
For illustration purposes, we used grid search for the \(\text {fPd}(\alpha , \mu )\) as it is relatively fast due to \(\alpha \) being bounded in (0, 1) and to \(\mu \), which is just in the neighborhood of the true data mean scaled by \(\Gamma (1 + \alpha ).\) Observe that \(5 \times 10^5\) random numbers are used in all the calculations. From the results below, the fractional Poisson distribution \(\text {fPd}(\alpha , \mu )\) provides better fits than the negative binomial \(\text {NegBinom}(\textit{size}, mean)\) model for both data sets at \(5 \%\) level of significance. This exercise clearly demonstrates the limitation of the negative binomial in dealing with left-skewed count data (Table 1).
2.2 The case for gfPd\((\alpha , \alpha , 1, \mu )\)
When \(\beta =\alpha \) and \(\delta =1\), we have \(X_{\alpha , \alpha } \; {\mathop {=}\limits ^{d}} \; \)gfPd\((\alpha , \alpha , 1, \mu )\) with
Proposition 2.1
The probability mass function can be written as
where \(\nu _S\) is the distribution of a random variable S whose density has Laplace transform \(\exp (-t^\alpha )\).
Proof
Note that
\(\square \)
The above result provides an algorithm to evaluate the probability mass function as
Thus, we can now estimate \(\alpha \) and \(\mu \) using maximum likelihood just like in the fPd case. The maximum likelihood estimates for the two crime datasets above are given in Table 2 below. The chi-square goodness-of-fit test statistics are large, indicating bad fits.
Remark 2.3
From (2.1), the random variable \(X_{\alpha ,\alpha }\) has factorial moments
Thus the pgf is \(G_{X_{\alpha ,\alpha }}(u)= \Gamma (\alpha ) E_{\alpha ,\alpha }^1 \left( \mu \left( u-1\right) \right) \), \(|u| \le 1\).
From (4) and (37), the symmetry structure of \(X_{\alpha , \alpha }\) can be determined as follows:
Moreover,
which vanishes if \(\alpha =1\) (Poisson distribution). Moreover, (39) is non-negative and decreasing: this explains the bad fits indicated by the large chi-square values above.
3 Underdispersion and overdispersion for weighted Poisson distributions
Weighted Poisson distributions (Rao 1965) provide a unifying approach for modelling both overdispersion and underdispersion (Kokonendji et al. 2008). Let Y be a Poisson random variable of parameter \(\lambda >0\) and let \(Y^{w}\) be the corresponding WPD with weight function w.
Theorem 3.1
If \({\varvec{E}}w(Y+k)<\infty \) for all \(k \in {\mathbb {N}}\), and \(a_k=\lambda ^k h(\lambda ,k)\), where \(h(\lambda ,k)=\frac{ {\varvec{E}}w(Y+k)}{ {\varvec{E}}w(Y)}\), satisfies (5), then \(Y^{w}\) has factorial moments \(a_k\).
Proof
It is enough to observe that the pgf \(G_{Y^{w}}(u)\) can be written in form (2) as follows:
\(\square \)
Let T be the linear left-shift operator acting on number sequences. Let us still denote with T its coefficientwise extension to the ring of formal power series in \({\mathbb R}_+[[\lambda ]]\) (Stanley 2012). Next proposition links overdispersion and underdispersion of \(Y^w\) respectively to a Turán-type and a reverse Turán-type inequality involving T.
Theorem 3.2
The random variable \(Y^w\) is overdispersed (underdispersed) if and only if
where \(f(\lambda ) = {\varvec{E}}w(Y)\).
Proof
The random variable \(Y^w\) is overdispersed if and only if \(a_2 > a_1^2\), that is \({\varvec{E}}w(Y){\varvec{E}}w(Y+2)>[ {\varvec{E}}w(Y+1)]^2.\) Equivalently,
and the result follows observing that \(T^j f(\lambda ) = \sum _{k \ge 0} \frac{\lambda ^k}{k!} T^j[ w(k)]\) for \(j=1,2\). \(\square \)
Remark 3.1
Observe that when w does not depend on \(\lambda \), then \(T^j f(\lambda ) = D_{\lambda }^j f(\lambda )\) for \(j=1,2.\) In this case, condition (41) is equivalent to \(f(\lambda ) D^2_{\lambda }f(\lambda )> (<)\,[D_{\lambda }f(\lambda )]^2\), i.e. log-convexity (log-concavity) of f. This is already known in the literature (see Theorem 3 of Kokonendji et al. (2008)).
Remark 3.2
Note that from (42) we have
and some algebra leads us to the following sufficient condition for overdispersion or underdispersion: the random variable \(Y^w\) is overdispersed (underdispersed) if
Notice that \({\varvec{E}}w(Y)\) is a function of the Poisson parameter \(\lambda \). For the sake of clarity, from now on, let us denote it by \(\eta (\lambda )\). Weighted Poisson distributions with a weight function w not depending on the Poisson parameter \(\lambda \) are also known as power series distributions (PSD) (Johnson et al. 2005) and it is easy to see that the factorial generating function in this case reads
with factorial moments
A special well-known family of PSD is the generalized hypergeometric probability distribution (GHPD) (Kemp 1968), where
with \(_pF_q\) given in (9). Depending on the values of the parameters of GHPD both overdispersion and underdispersion are possible (Tripathi and Gurland 1979). For \(p=q=1,\) a special case of GHPD is the hyper-Poisson distribution (Bardwell and Crow 1964). In the next section we will analyze an alternative WPD in which the hyper-Poisson distribution remains a special case and that exhibits both underdispersion and overdispersion.
3.1 A novel flexible WPD allowing overdispersion or underdispersion
Let \(Y^w\) be a WP random variable with weight function
where \(\gamma > 0\), \( \min (\alpha , \beta ,\nu )\ge 0\), \(\alpha +\beta >0\). Moreover, if \(\gamma =\beta \) and \(\nu \ge 1\) then \(\beta \) is allowed to be zero. Since it is a PSD, the random variable \(Y^w\) is characterized by the normalizing function
The convergence of the above series can be ascertained as follows. Let \(\gamma \le 1\); by Gautschi’s inequality (see Qi (2010), formula (2.23)) we have the upper bound
which converges by ratio test and taking into account the well-known asymptotics for the ratio of gamma functions (see Tricomi and Erdélyi (1951)). Now, let \(\gamma > 1\). In this case an upper bound can be derived by formula (3.72) of Qi (2010):
Again, this converges by ratio test and recurring to the above-mentioned asymptotic behaviour of the ratio of gamma functions.
The random variable \(Y^w\) specializes to some well-known classical random variables. Specifically, we recognize the following:
-
1.
If \(\gamma =\beta =\alpha =\nu =1\), we recover the Poisson distribution as the weights equal unity for each k.
-
2.
If \(\gamma =\beta =\alpha =1\), we recover the COM-Poisson distribution (Conway and Maxwell 1962) of Poisson parameter \(\lambda \) and dispersion parameter \(\nu \).
-
3.
If \(\gamma =\alpha =\nu =1\) we obtain the hyper-Poisson distribution (Bardwell and Crow 1964).
-
4.
If \(\gamma =\nu =1\) we obtain the alternative Mittag–Leffler distribution considered e.g. in Bardwell and Crow (1964) and Herrmann (2016).
-
5.
If \(\gamma =1\) we recover the fractional COM-Poisson distribution (Garra et al. 2018).
-
6.
If \(\nu =1\) we obtain the alternative generalized Mittag–Leffler distribution (Pogány and Tomovski 2016).
Since \(Y^w\) is a PSD, it is easy to derive its factorial moments,
from which the moments are immediately derived by recalling formula (3).
Remark 3.3
Since \(\eta _{\alpha ,\alpha r+\beta }^{\gamma +r,\nu }(\lambda ) = \sum _{j \ge 0} \frac{\lambda ^j}{j!} A_{j,r}\) with
by using Faà di Bruno’s formula (Stanley 2012) one has
where \(\{A_{j,0}\}\) and and \(\{{{\mathfrak {B}}}_{i,k}\}\) are the coefficients of \(\eta _{\alpha , \beta }^{\gamma ,\nu }(\lambda )\) and the partial Bell exponential polynomials (Stanley 2012), respectively.
Furthermore, the probability mass function reads
Concerning the variability of \(Y^w\), by using Theorem 3 of Kokonendji et al. (2008), the preceding Lemma and the succeeding Corollary, that is by imposing log-convexity (log-concavity) of the weight function, we write for \(y \in {\mathbb {R}}_+\),
where \(\psi (z)\) is the Psi function (see Lebedev (1972), Section 1.3). In addition, by considering formula (6.4.10) of Abramowitz and Stegun (1964),
Therefore log-convexity (log-concavity) of w(y) is equivalent to the condition
This yields that if (58) holds, then \(Y^w\) is overdispersed (underdispersed).
Remark 3.4
(Classical special cases) If \(\alpha =\beta =\gamma =1\), then \(Y^w\) is the COM-Poisson random variable and (58) correctly reduces to the ranges \(\nu >1\) giving underdispersion and \(\nu \in [0,1)\) giving overdispersion. If \(\alpha =\gamma =\nu =1\), then \(Y^w\) is the hyper-Poisson random variable and (58) correctly reduces to the ranges \(\beta >1\) (overdispersion) and \(\beta \in [0,1)\) (underdispersion). This holds as \(\beta \mapsto \sum _{r=0}^\infty (y+\beta +r)^{-2}\) is decreasing for all fixed \(y \in {\mathbb {R}}_+\).
In the two next sections we analyze two special cases of interest, the first of which, to the best of our knowledge, is still not considered in the literature.
3.1.1 Model I
We first introduce the special case in which \(\alpha =1\), \(\gamma =\beta \), \(\beta >0\), and \(\beta \) is allowed to be zero only if \(\nu \ge 1\). This is a three-parameter (\(\lambda ,\nu ,\beta \)) model which retains the same simple conditions for underdispersion and overdispersion as for the COM-Poisson model. Indeed, formula (58) reduces to \(\nu >1\) and \(\nu \in [0,1)\), respectively. However, this model is more flexible than the COM-Poisson model because of the presence of the parameter \(\beta \). Notice that the pmf can be written as
which suggests that Model I belongs to the exponential family of distributions with parameters \(\log \lambda \) and \(1-\nu \), where \(\beta \) is a nuisance parameter or is known. Figures 4 and 5 show sample shapes of this family of distributions.
Note that distributions in Fig. 4 (Fig. 5) are overdispersed (underdispersed). Also,
This gives a procedure to calculate iteratively the probability mass function and generate random numbers. The only thing to figure out is to compute \(\eta _{1,\beta }^{\beta ,\nu }(\lambda )\) in order to obtain \(P(Y^w = 0)=1/[\Gamma (\beta )^{\nu -1} \eta _{1,\beta }^{\beta ,\nu }(\lambda )]\).
An upper bound for the normalizing function \(\eta _{1,\beta }^{\beta ,\nu }(\lambda )\) can be determined similarly to Minka et al. (2003), Section 3.2, taking into consideration that the multiplier
is ultimately monotonically decreasing. Hence, we can approximate the normalizing constant \(\eta _{1,\beta }^{\beta ,\nu }(\lambda )\) by truncating the series and bound the truncation error \(R_{{\widetilde{k}}}\),
where \({\widetilde{k}}\) is such that for \(j>{\widetilde{k}}\) the multiplier (61) is already monotonically decreasing and bounded above by \(\varepsilon _{{\widetilde{k}}} \in (0,1)\). Correspondingly, denoting with \({\widetilde{\eta }}_{1,\beta }^{\beta ,\nu }(\lambda ) = \sum _{j=0}^{{\widetilde{k}}} \frac{\lambda ^j}{j!} \Gamma (j+\beta )^{1-\nu }\), the relative truncation error \(R_{{\widetilde{k}}}/{\widetilde{\eta }}_{1,\beta }^{\beta ,\nu }(\lambda )\) is bounded by
As a last remark, we can further simplify the model obtaining a two-parameter model. In order to do so, let \(\nu =\beta \), with \(\beta >0\). The obtained model still allows for underdispersion (\(\beta >1\)) and overdispersion (\(\beta \in (0,1)\)) and it should be directly compared with the COM-Poisson and the hyper-Poisson models.
3.1.2 Model II
If we set \(\alpha =\nu =1\) we get another three-parameter (\(\lambda ,\gamma ,\beta \)) model, special case of the alternative generalized Mittag–Leffler distribution (see point 6 above). The reparametrization \(\beta =\xi \gamma \) together with condition (58) shows that both overdispersion (\(\xi >1\)) and underdispersion (\(\xi \in (0,1)\)) are possible. This comes from the fact that \(\omega \mapsto \sum _{r=0}^\infty (y+\omega +r)^{-2}\) is decreasing for all fixed \(y \in {\mathbb {R}}_+\). As for Model I, the probability distribution belongs to the exponential family with parameter \(\log \lambda ,\) with \(\gamma \) and \(\beta \) as nuisance parameters. Explicitly, the pmf reads
and, as in the previous Sect. 3.1.1, the iterative representation
allows an approximated evaluation of the pmf with error control, and consequently random number generation. Also in this case this holds as the involved multiplier is ultimately monotonically decreasing. Figures 6 and 7 show some forms of this class of distributions. Observe that distributions in Fig. 6 (Fig. 7) are underdispersed (overdispersed).
If we further let \(\lambda =1\), we obtain a two-parameter model, still allowing for underdispersion if \(\beta \in (0,\gamma )\) (or equivalently \(\xi \in (0,1)\)) and overdispersion if \(\beta > \gamma \) (or \(\xi > 1\)), which is also directly comparable with the two-parameter Model I above, the COM-Poisson model, and the hyper-Poisson model.
3.1.3 Comparison
We now compare Model I and Model II with known models that allow overdispersion and underdispersion such as the COM-Poisson, generalized Poisson and hyper-Poisson models as cited above. Note that the hyper-Poisson distribution satisfies
For comparison purposes, we first consider the number of fish caught dataFootnote 2 shown in Fig. 8 (left panel) below. The dataset corresponds to 239 groups (as 11 potential outliers were removed) that went to a state park and state wildlife biologists asked visitors how many fish they caught. The mean fish caught is around 1.48 while the variance is 8.04. Furthermore, the optimx (for hyper-Poisson, Model I and Model II), COMPoissonReg (for COM-Poisson), compoisson (for COM-Poisson), and VGAM (for generalized Poisson) packages in R are used for the maximum likelihood estimation and the chi-square goodness-of-fit tests. In particular, the L-BFGS-B method from the optimx package is used and 1000 terms were summed for the normalizing constant \(\eta _{\alpha ,\beta }^{\gamma , \nu } (\lambda )\). Just like the comparisons above, a chi-square distribution is used as reference where the degrees of freedom is the number of cells minus the number of model parameters. From Table 3, Model I and Model II clearly outperform the other models although the generalized Poisson and hyper-Poisson (subcase of WPD) also provide good fits to the fish count data.
We have also considered the bioChemists data from the pscl package in R, particularly the count of articles produced by 915 graduate students in biochemistry Ph.D. programs during last 3 years in the program. The data has mean 1.69 and variance of 3.71, and is showcased in Fig. 8 (right panel). Apparently, Table 4 suggests that Model II outperforms the rest of the models considered for the article count data. Overall, there is potential in WPD’s (e.g., Model I and Model II) in flexibly capturing overdispersed and/or underdispersed count data distributions.
References
Abramowitz M, Stegun IA (1964) Handbook of mathematical functions with formulas, graphs, and mathematical tables. Natl Bureau Stand Appl Math Ser 55:1046
Bardwell GE, Crow EL (1964) A two-parameter family of hyper-Poisson distributions. J Am Stat Assoc 59:133–141
Beghin L, Orsingher E (2009) Fractional Poisson processes and related planar random motions. Electron J Probab 14:1790–1826
Beghin L, Orsingher E (2010) Poisson-type processes governed by fractional and higher-order recursive differential equations. Electron J Probab 15(22):684–709
Cahoy DO, Uchaikin VV, Woyczynski WA (2010) Parameter estimation for fractional Poisson processes. J Stat Plan Inference 140(11):3106–3120
Chambers JM, Mallows CL, Stuck BW (1976) A method for simulating stable random variables. J Am Stat Assoc 71(354):340–344
Consul PC, Jain GC (1973) A generalization of the Poisson distribution. Technometrics 15:791–799
Conway RW, Maxwell WL (1962) A queuing model with state dependent service rates. J Ind Eng 12:132–136
Daley DJ, Narayan P (1980) Series expansions of probability generating functions and bounds for the extinction probability of branching process. J Appl Probab 17(4):939–947
De Oliveira EC, Mainardi F, Vaz J (2011) Models based on Mittag-Leffler functions for anomalous relaxation in dielectrics. Eur Phys J Spec Top 193(1):161–171
Di Nardo E, Senato D (2006) An umbral setting for cumulants and factorial moments. Eur J Comb 27:394–413
Garra R, Orsingher E, Polito F (2018) A Note on Hadamard fractional differential equations with varying coefficients and their applications in probability. Mathematics 6(1), 4, 10pp
Herrmann R (2016) Generalization of the fractional Poisson distribution. Fract Calc Appl Anal 19(4):832–842
Johnson NL, Kemp AW, Kotz S (2005) Univariate discrete distributions, IIIrd edn. Wiley Series in Probability and Statistics. Wiley, New York
Jumarie G (2012) Fractional master equation: non-standard analysis and Liouville-Riemann derivative. Chaos Solut Fractals 12:2577–2587
Kanter M (1975) Stable densities under change of scale and total variation inequalities. Ann Probab 3(4):697–707
Kemp AW (1968) A wide class of discrete distributions and the associated differential equations. Sankhya (A) 30:401–410
Kemp A, Kemp C (1974) A family of discrete distributions defined via their factorial moments. Commun Stat 3(12):1187–1196
Kilbas AA, Srivastava HM, Trujillo JJ (2006) Theory and applications of fractional differential equations. North-Holland Mathematics Studies, vol 204. Elsevier, Amsterdam
Kokonendji CC, Mizère D, Balakrishnan N (2008) Connections of the Poisson weight function to overdispersion and underdispersion. J Stat Plan Inference 138(5):1287–1296
Laskin N (2003) Fractional Poisson process. Commun Nonlinear Sci Numer Simul 8:201–213
Lebedev NN (1972) Special functions and their applications. Dover Publications Inc, New York
Maceda EC (1948) On the compound and generalized Poisson distributions. Ann Math Stat 19:414–416
Mainardi F, Mura A, Pagnini G (2010) The \(M\)-Wright function in time-fractional diffusion processes: a tutorial survey. International Journal of Differential Equations, Art. ID 104505, p 29
Meerschaert MM, Nane E, Vellaisamy P (2011) The fractional Poisson process and the inverse stable subordinator. Electron J Probab 16:1600–1620
Minka TP, Shmueli G, Kadane JB, Borle S, Boatwright P (2003) Computing with the COM-Poisson distribution. Technical Report 775. Department of Statistics, Carnegie Mellon University, Pittsburgh. (Available from http://www.stat.cmu.edu/tr/)
Polito F, Tomovski Ž (2016) Some properties of Prabhakar-type fractional calculus operators. Fract Differ Calc 6(1):73–94
Potts RB (1953) Note on the factorial moments of standard distributions. Aust J Phys 6(4):498–499
Prabhakar TR (1971) A singular integral equation with a generalized Mittag-Leffler function in the kernel. Yokohama Math J 19:7–15
Rao CR (1965) On discrete distributions arising out of methods of ascertainment. Sankhyā (A) 27:311–324
Saichev AI, Zaslavsky GM (1997) Fractional kinetic equations: solutions and applications. Chaos 7(4):753–764
Shmueli G, Minka TP, Kadane J, Borle S, Boatwright P (2005) A useful distribution for fitting discrete data: revival of the Conway-Maxwell-Poisson distribution. J R Stat Soc Ser C 54(1):127–142
Repin ON, Saichev AI (2000) Fractional Poisson law. Radiophys Quant Electron 43(9):738–741
Stanley RP (2012) Enumerative combinatorics. vol 1, Cambridge Studies in Advanced Mathematics, 49, Cambridge University Press
Pogány TK, Tomovski Z (2016) Probability distribution built by Prabhakar function. Related Turán and Laguerre inequalities. Integral Transforms Special Funct 27(10):783–793
Qi F (2010) Bounds for the ratio of two gamma functions. J Inequal Appl. Art. ID 493058, 84 pp
Tripathi RC, Gurland J (1979) Some aspects of the Kemp families of distributions. Commun Stat A 8:855–869
Tricomi FG, Erdélyi A (1951) The asymptotic expansion of a ratio of gamma functions. Pac J Math 1:133–142
Acknowledgements
F. Polito has been partially supported by the project “Memory in Evolving Graphs” (Compagnia di San Paolo/Università degli Studi di Torino).
Funding
Open Access funding provided by Università degli Studi di Torino.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Cahoy, D., Di Nardo, E. & Polito, F. Flexible models for overdispersed and underdispersed count data. Stat Papers 62, 2969–2990 (2021). https://doi.org/10.1007/s00362-021-01222-7
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00362-021-01222-7