Bias Correction in Estimating Proportions by Imperfect Pooled Testing

Abstract

In the estimation of proportions by pooled testing, the MLE is biased. Hepworth and Biggerstaff (JABES, 22:602–614, 2017) proposed an estimator based on the bias correction method of Firth (Biometrika 80:27–38, 1993) and showed that it is almost unbiased across a range of pooled testing problems involving no misclassification. We now extend their work to allow for imperfect testing. We derive the estimator, provide a Newton–Raphson iterative formula for its computation and test it in situations involving equal or unequal pool sizes, drawing on problems encountered in plant disease assessment and prevalence estimation of mosquito-borne viruses. Our estimator is highly effective at reducing the bias for prevalences consistent with the pooled testing procedure employed.

Appendices

Appendix 1: Derivation of Firth’s Bias Correction Applied to Imperfect Pooled Testing

Recall $\pi (p)= a - r(1-p)^m$. The first two derivatives are

$$\begin{aligned} \frac{\text{ d }\pi (p)}{\text{ d }p}= & {} mr(1-p)^{m-1} = \frac{m}{1-p} r(1-p)^m = \frac{m}{1-p} \left[ a-\pi (p)\right] , \\ \frac{\text{ d}^2\pi (p)}{\text{ d }p^2}= & {} -m(m-1)r(1-p)^{m-2} = -\frac{m(m-1)}{(1-p)^2} \left[ a-\pi (p)\right] . \end{aligned}$$

From here on, we drop the functional notation on $\pi (p)$. We need the following derivatives of the log-likelihood:

$$\begin{aligned} \frac{\text{ d }\,l}{\text{ d }\pi }= & {} \frac{x}{\pi } - \frac{n-x}{1-\pi }, \\ \frac{\text{ d}^2 l}{\text{ d }\pi ^2}= & {} -\frac{x}{\pi ^2} - \frac{n-x}{(1-\pi )^2}, \\ \frac{\text{ d}^3 l}{\text{ d }\pi ^3}= & {} \frac{2x}{\pi ^3} - \frac{2(n-x)}{(1-\pi )^3}. \end{aligned}$$

From these, we have the score function

$$\begin{aligned} S(p) = \frac{\text{ d }\,l}{\text{ d }p}= & {} \frac{\text{ d }\,l}{\text{ d }\pi } \frac{\text{ d }\pi }{\text{ d }p} = \frac{m (a-\pi ) (x - n\pi )}{(1-p)\pi (1-\pi )}. \end{aligned}$$

(A.1)

The chain rule and product rule can be used to compute the higher-order derivatives of the likelihood, as follows:

$$\begin{aligned} \frac{\text{ d}^2 l}{\text{ d }p^2}= & {} \frac{\text{ d}^2 l}{\text{ d }\pi ^2} \left( \frac{\text{ d }\pi }{\text{ d }p}\right) ^2 + \; \frac{\text{ d }\,l}{\text{ d }\pi }\, \frac{\text{ d}^2\pi }{\text{ d }p^2}, \\ \frac{\text{ d}^3 l}{\text{ d }p^3}= & {} \frac{\text{ d}^3 l}{\text{ d }\pi ^3} \left( \frac{\text{ d }\pi }{\text{ d }p}\right) ^3 + 3\, \frac{\text{ d}^2 l}{\text{ d }\pi ^2}\, \frac{\text{ d}^2 \pi }{\text{ d }p^2}\, \frac{\text{ d }\pi }{\text{ d }p} + \frac{\text{ d }\,l}{\text{ d }\pi }\, \frac{\text{ d}^3 \pi }{\text{ d }p^3}. \end{aligned}$$

We therefore can obtain the information I(p) as:

$$\begin{aligned} I(p)= & {} E\left[ -\frac{\text{ d}^2 l}{\text{ d }p^2}\right] \\= & {} E\left[ -\left( -\frac{x}{\pi ^2} - \frac{n-x}{(1-\pi )^2}\right) \left( \frac{m}{1-p}(a-\pi )\right) ^2 - \left( \frac{x}{\pi } - \frac{n-x}{1-\pi }\right) \left( -\frac{m(m-1)(a-\pi )}{(1-p)^2}\right) \right] \\= & {} \left( \frac{n}{\pi } + \frac{n}{1-\pi }\right) \left( \frac{m}{1-p}(a-\pi )\right) ^2\\= & {} \frac{n m^2 (a-\pi )^2}{(1-p)^2 \pi (1-\pi )}. \end{aligned}$$

Computation of the bias (see Eq. 3) requires $\text{ d }I(p) / \text{ d }p$ and $E\left[ \text{ d}^3 l / \text{ d }p^3\right]$ in addition to I(p). The derivation of $\text{ d }I(p) / \text{ d }p$ is more tedious than for perfect testing, because our expression for I(p) includes both p and $\pi .$ We remedy this (i.e., put it in terms of $\pi$ alone and p only implicitly) by writing

$$\begin{aligned} I(p)= & {} \frac{nm^2(a-\pi )^2}{(1-p)^2\pi (1-\pi )} = \frac{nm^2(a-\pi )^2}{\left( \frac{a-\pi }{r}\right) ^\frac{2}{m} \pi (1-\pi )}\,. \end{aligned}$$

(A.2)

Then

$$\begin{aligned} \frac{\text{ d }I(p)}{\text{ d }p}= & {} \frac{\text{ d }I(p)}{\text{ d }\pi } \frac{\text{ d }\pi }{\text{ d }p} \nonumber \\= & {} nm^2 \frac{\left( \frac{a-\pi }{r}\right) ^\frac{2}{m} \pi (1-\pi ) [-2(a-\pi )] - (a-\pi )^2\left[ \left( \frac{a-\pi }{r}\right) ^\frac{2}{m} (1-2\pi ) + \pi (1-\pi ) \frac{2}{m} \left( \frac{a-\pi }{r}\right) ^\frac{2-m}{m}\left( -\frac{1}{r}\right) \right] }{ \left( \frac{a-\pi }{r}\right) ^\frac{4}{m} \pi ^2 (1-\pi )^2} \nonumber \\&\times \left[ \frac{m}{1-p} (a - \pi )\right] \nonumber \\= & {} -\frac{nm^2(a-\pi )^2}{(1-p)^3\pi ^2(1-\pi )^2}\left[ 2(m-1)\pi (1-\pi ) + m(a-\pi )(1-2\pi )\right] \end{aligned}$$

(A.3)

noting that $\frac{r}{a-\pi } = (1-p)^{-m}$ is useful during the simplification. The final quantity for computing the bias b(p) is

$$\begin{aligned} E\left[ \frac{\text{ d}^3 l}{\text{ d }p^3}\right]= & {} E\left[ \left( \frac{2x}{\pi ^3} - \frac{2(n-x)}{(1-\pi )^3}\right) \left( \frac{m}{1-p}(a-\pi )\right) ^3 \right. \\&+ 3\, \left( -\frac{x}{\pi ^2} - \frac{n-x}{(1-\pi )^2}\right) \left( \frac{m}{1-p}(a-\pi )\right) \left( -\frac{m(m-1)(a-\pi )}{(1-p)^3}\right) \\&+\left. \left( \frac{x}{\pi } - \frac{n-x}{1-\pi }\right) \left( \frac{m(m-1)(m-2)(a-\pi )}{(1-\pi )^3}\right) \right] \\= & {} \left( \frac{2n}{\pi ^2} - \frac{2n}{(1-\pi )^2}\right) \left( \frac{m}{1-p}(a-\pi )\right) ^3 + 3\,\left( \frac{n}{\pi } + \frac{n}{1-\pi }\right) \left( \frac{m^2(m-1)(a-\pi )^2}{(1-p)^3}\right) \\= & {} \frac{n m^2 (a-\pi )^2}{(1-p)^3\pi ^2(1-\pi )^2} \left[ 2m(1-2\pi )(a-\pi ) + 3(m-1)\pi (1-\pi ])\right] . \end{aligned}$$

Further simplification results in the following expression for the numerator of the bias (see Eq. 3):

$$\begin{aligned} 2\frac{\text{ d }I(p)}{\text{ d }p} + E\left[ \frac{\text{ d}^3 l}{\text{ d }p^3}\right]= & {} -\frac{nm^2(m-1)(a-\pi )^2}{(1-p)^3\pi (1-\pi )}. \end{aligned}$$

(A.4)

Appendix 2: R Code for Newton–Raphson Iteration to Find Firth’s Bias-Corrected Estimate of ${{\varvec{p}}}$