Estimating the number of sequencing errors in microbial diversity studies

Abstract

Species diversity analysis of microbial communities is an important tool for assessing an ecosystem health. The advent of high-throughput genome sequencing techniques has made it possible to process an unprecedented number of RNA sequences. However, many studies report the presence of a significant number of fictitious rare species in datasets generated using these techniques. These species are the product of errors that can occur at any step of the sequence analysis pipeline. The overcount of rare species (especially singletons) affects the estimation of the total number of species, and of the diversity of the community as measured by Shannon’s index. To avoid overestimating these quantities, it is crucial to model the source of error. In this work, we present a new model that treats spurious singletons as false-negative record linkage errors, and compare it with another approach where spurious singletons are considered for deletion. We discuss the two inferential approaches both with an application to real data and on theoretical grounds. We demonstrate that, while Shannon’s index can differ significantly under the two models, the estimate of the total number of species is equivalent.

Appendix: conditional ABC algorithm

In this Appendix we give some details on the ABC rejection sampler conditioned on the number of captures s presented in Sect. 5.2.1. As we have noted there, to condition ourselves on the total number of specimens s, which remains fixed in the hypothesis of the MLM, we have to replace the first two steps of the naive algorithm presented in Sect. 5.2, with the following two:

• 1. generate values for $(\theta ,N^*)$ given s and the priors $\pi (\theta )$ and $\pi (N^*)$

• 2. generate values $(n_0^*,n_1^*,n_2^*,...)$ conditional on $N^*$, $\theta$ and s

In the following three subsections, we detail the passages for the two steps above according to the chosen baseline.

1.1 Poisson

If we consider a $Poi(\lambda )$ baseline distribution for our MLM, we have

$$\begin{aligned} \sum X_i^* \,|\; \lambda ,N^* \; \sim \; Poi(\lambda N^*). \end{aligned}$$

and, by integrating over $\lambda$ with prior $Gamma(\alpha _{\lambda }, \beta _{\lambda })$ we have

$$\begin{aligned} \sum X_i^* \,|\; N^* \; \sim \; NegBin\left( \alpha _{\lambda },\frac{\beta _{\lambda }}{N^*+\beta _{\lambda }}\right) . \end{aligned}$$

Given a prior in the family $\pi (N^*) \propto (N^*){}^{-k}$ defined by k, we can calculate the probability $P(N^* \,|\, s )$ for a sufficiently large range of values, via

$$\begin{aligned} P\left( N^* \,|\, \textstyle \sum X^*_i = s \right)&\propto P\left( \textstyle \sum X^*_i = s \,|\, N^* \right) \pi (N^*), \nonumber \\&\propto \exp {\left\{ (s-k)\log (N^*) - (\alpha _{\lambda }+s) \log (N^*+\beta _{\lambda }) \right\} }. \end{aligned}$$

(9)

Then, step 1 amounts to generate a value for $N^* | s$ according to (9), and then generate $\lambda$ from $P( \lambda \,|\, N^*, s )$, which is the updated Gamma distribution

$$\begin{aligned} Gamma\left( \alpha _{\lambda } + s \,,\, \beta _{\lambda }+N^*\right) . \end{aligned}$$

As for step 2. of our scheme, note that the distribution of $(n_0^*,n_1^*,n_2^*,...)$ conditional on $N^*$ and s is independent of $\lambda$. In fact, it is well-known that the joint distribution of $N^*$ independent Poisson having fixed sum s is Multinomial with uniform probabilities:

$$\begin{aligned} \left( X^*_1,...,X^*_{N^*}\,|\, \textstyle \sum X^*_i = s\right) \sim Mult\big (s, (1/N^*,...,1/N^*)\big ). \end{aligned}$$

So, we can generate $(x_1^*,...,x^*_{N^*})$ having fixed sum s directly from the Multinomial above.

1.2 Geometric

Under a Geo(p) baseline distribution we have

$$\begin{aligned} \sum X_i^* \,|\; p,N^* \sim NegBin(N^*,p). \end{aligned}$$

Then, if we adopt a prior $p \sim Beta(\alpha _p, \beta _p)$, by integrating out p, we have

$$\begin{aligned} P\left( \sum X_i^*=s \,|\, N^*\right) = \left( {\begin{array}{c}N^*+s-1\\ s\end{array}}\right) \frac{B(\alpha _p+N^*,\beta _p+s)}{B(\alpha _p,\beta _p)}, \end{aligned}$$

where B denotes the Beta function. Thus, we can calculate the probability $P(N^* \,|\, s)$ for a sufficiently large range of values as:

$$\begin{aligned} P(N^* \,|\, s) \propto \frac{\Gamma (N^*+s) \Gamma (\alpha _p+N)}{\Gamma (N^*) \Gamma (\alpha _p+\beta _p+N^*+s) } \cdot \pi (N^*), \end{aligned}$$

(10)

and generate values for $N^*$ accordingly. Then, we can generate values for p from $P(p \,|\, N^*,s)$, which is the updated Beta distribution:

$$\begin{aligned} Beta(\alpha _p + N^*, \beta _p+s). \end{aligned}$$

The distribution of $(n_0^*,n_1^*,n_2^*,...)$ conditional on $N^*$ and s is independent of p. In fact, all possible vectors $(x_1^*,...,x^*_{N^*})$ with fixed sum s have the same probability $\left( {\begin{array}{c}N^*+s-1\\ s\end{array}}\right) ^{-1}$, equal to the reciprocal of the number of possible nonnegative integer $N^*$-vectors summing to s (or weak $N^*$–compositions of s). As a consequence, step 2. of our scheme can be completed by using an algorithm for random compositions,(see, e.g., Nijenhuis and Wilf (1978) or Stojmenović (1992)), to generate a nonnegative integer vector $(x_1^*,...,x^*_{N^*})$ with fixed sum s.

1.3 Negative Binomial

Under a NegBin(r, p) baseline distribution we have:

$$\begin{aligned} P\left( \sum _{i=1}^{N^*} X_i^* = s \,|\, r,p,N^*\right) = \left( {\begin{array}{c}rN^*+s-1\\ s\end{array}}\right) p^{rN^*} (1-p)^s. \end{aligned}$$

(11)

In order to generate values from the posterior of $(N^*,r,p)$ given s, we can generate values from their (independent) priors and accept them with probability given by (11).

Consider the following result (see, e.g., Guimaraes and Lindrooth 2007):

Proposition 3

Let $X^*_i \sim NegBin(r,p)$, $i=1,...,N^*$, and let $\sum _{i=1}^{N^*}X^*_i=s$ then:

$$\begin{aligned} P(x_1^*,...,x^*_{N^*} | s) = \frac{\Gamma (N^* r) \Gamma (s+1)}{\Gamma (s+N^* r)} \prod _{i=1}^{N^*} \frac{\Gamma (x^*_i +r)}{\Gamma (r) \Gamma (x^*_i+1)}. \end{aligned}$$

Then, for the second step, we generate $x^*_1,...,x^*_{N^*}$ with fixed sum s from the Dirichlet Multinomial defined in Proposition 3.