A simple method for implementing Monte Carlo tests

We consider a statistical test whose p-value can only be approximated using Monte Carlo simulations. We are interested in deciding whether the p-value for an observed data set lies above or below a given threshold such as 5%. We want to ensure that the resampling risk, the probability of the (Monte Carlo) decision being different from the true decision, is uniformly bounded. This article introduces a simple open-ended method with this property, the confidence sequence method (CSM). We compare our approach to another algorithm, SIMCTEST, which also guarantees an (asymptotic) uniform bound on the resampling risk, as well as to other Monte Carlo procedures without a uniform bound. CSM is free of tuning parameters and conservative. It has the same theoretical guarantee as SIMCTEST and, in many settings, similar stopping boundaries. As it is much simpler than other methods, CSM is a useful method for practical applications.


Introduction
Suppose we want to use a one-sided statistical test with null hypothesis H based on a test statistic T with observed value t. We aim to calculate the p-value where the measure P is ideally the true null distribution in a simple hypothesis. Otherwise, it can be an estimated distribution in a bootstrap scheme, or a distribution conditional on an ancillary statistic, etc.
We consider the scenario in which p cannot be evaluated in closed form, but can be approximated using Monte Carlo simulation, e.g. through bootstrapping or drawing permutations. To be precise, we assume we can generate a stream (T i ) i∈N of i.i.d. random variables from the distribution of a test statistic T under P. The information about whether or not T i exceeds the observed value t is contained in the random variable X i = I(T i ≥ t), where I denotes the indicator function. It is a Bernoulli random variable satisfying P(X i = 1) = p. We will formulate algorithms in terms of X i . Gleser (1996) suggests that two individuals using the same statistical method on the same data should reach the same conclusion. For tests, the standard decision rule is based on comparing p to a threshold α. In the setting we consider, Monte Carlo methods are used to compute an estimatep of p, which is then compared to α to reach a decision.
We are interested in procedures which provide a user-specified uniform bound > 0 on the probability of the decision based onp being different to the decision based on p. We call this probability the resampling risk and define it formally as wherep is the p-value estimate computed by a Monte Carlo procedure. We are looking for procedures that achieve sup for a pre-specified > 0. We introduce a simple sequential Monte Carlo testing procedure achieving (1) for any > 0, which we call the confidence sequence method (CSM). Our method is based on a confidence sequence for p, that is a sequence of (random) intervals with a joint coverage probability of at least 1 − . We will use the sequences constructed in (Robbins, 1970;Lai, 1976). A decision whether to reject H is reached as soon as the first interval produced in the confidence sequence ceases to contain the threshold α.
The basic (non-sequential) Monte Carlo estimator (Davison and Hinkley, 1997) p = 1 + n i=1 X i 1 + n does not guarantee a small uniform bound on RR p (p), where n is a pre-specified number of Monte Carlo simulations. In fact, the lowest uniform bound on the resampling risk for this estimator is at least 0.5 (Gandy, 2009).
A variety of procedures for sequential Monte Carlo testing are available in the literature which target different error measures. Silva et al. (2009) ;Silva and Assunção (2013) bound the power loss of the test while minimising the expected number of steps. Silva and Assunção (2018, Section 4) construct truncated sequential Monte Carlo algorithms which bound the power loss and the level of significance in comparison to the exact test by arbitrarily small numbers. Other algorithms aim to control the resampling risk (Fay and Follmann, 2002;Fay et al., 2007;Gandy, 2009;Kim, 2010). Fay et al. (2007) use a truncated sequential probability ratio test (SPRT) boundary and discuss the resampling risk, but do not aim for a uniform lower bound on it. Fay and Follmann (2002) and Kim (2010) ensure a uniform bound on the resampling risk under the assumption that the random variable p belongs to a certain class of distributions. Besides being a much less restrictive requirement than (1), one drawback of this approach is that in real situations, the distribution of p is typically not fully known, as this would require knowledge of the underlying true sampling distribution.
We mainly compare our method to the existing approach of Gandy (2009), which we call SIMCTEST in the present article. SIMCTEST works on the partial sum S n = n i=1 X i and reaches a decision on p as soon as S n crosses suitably constructed decision boundaries. SIMCTEST is specifically constructed to guarantee a desired uniform bound on the resampling risk.
Procedures for Monte Carlo testing can be classified as open-ended and truncated procedures (Silva and Assunção, 2013). A truncated approach (Davidson and MacKinnon, 2000;Silva and Assunção, 2018;Besag and Clifford, 1991;Silva and Assunção, 2013) specifies a maximum number of Monte Carlo simulations in advance and forces a decision before or at the end of all simulations. Open-ended procedures (e.g., Gandy (2009)) do not have a stopping rule or an upper bound on the number of steps. Open-ended procedures can be turned into truncated procedures by forcing a decision after a fixed number of steps. Truncated procedures cannot guarantee a uniform bound on the resampling risk -Section 6 demonstrates this.
In practice, a truly open-ended procedure will often not be feasible. This is obvious in settings where generating samples is very time-consuming (Tango and Takahashi, 2005;Kulldorff, 2001), but it is also true in other settings where a large number of samples is being generated, as infinite computational effort is never available.
In the related context of Monte Carlo tests for multiple comparisons, algorithms guaranteeing a uniform error bound on any erroneous decision have been considered (Gandy andHahn, 2014, 2016). Though not explicitly discussed, CSM could in fact be considered a special case of the framework of Gandy and Hahn (2016) when testing one hypothesis using a Bonferroni correction (Bonferroni, 1936).
This article is structured as follows. We first describe the confidence sequence method in Section 2 before briefly reviewing SIMCTEST in Section 3. We compare the (implied) stopping boundaries of both methods and investigate the real resampling risk incurred from their use in Section 4. In Section 5, we investigate the rate at which both methods spend the resampling risk and construct a new spending sequence for SIMCTEST which (empirically) gives uniformly tighter stopping boundaries than CSM, thus leading to faster decisions on p.
In Section 6, we show that neither procedure bounds the resampling risk in the truncated Monte Carlo setting, but that the truncated version of CSM performs well compared to more complicated algorithms. The article concludes with an example application in Section 7 and a discussion in Section 8.

The Confidence Sequence Method
This section introduces the confidence sequence method (CSM), a simple algorithm to compute a decision on p while bounding the resampling risk.
Let ∈ (0, 1) be the desired bound on the resampling risk. Using independent Bernoulli(p) random variables X i , i ∈ N, the following inequality holds (Robbins, 1970): for all p ∈ (0, 1), where b(n, p, x) = n x p x (1 − p) n−x and S n = n i=1 X i . Then, I n = {p ∈ [0, 1] : (n+1)b(n, p, S n ) > } is a sequence of confidence sets that has the desired joint coverage probability 1 − .
The I n are intervals (Lai, 1976). Indeed, if 0 < S n < n we obtain I n = (g n (S n ), f n (S n )), where g n (x) < f n (x) are the two distinct roots of (n + 1)b(n, p, x) = . If S n = 0 then the equation (n+1)b(n, p, 0) = has only one root r n , leading to I n = [0, r n ). Likewise for S n = n, in which case I n = (r n , 1].
CSM will determine a decision on H as follows. We take samples until α / ∈ I n , leading to the stopping time τ = inf{n ∈ N : α / ∈ I n }.
If I n ⊆ [0, α] we reject H. If I n ⊆ (α, 1] we do not reject H. By construction, the uniform bound on the resampling risk in (1) holds true. It is not necessary to compute the roots of (n + 1)b(n, p, x) = explicitly to check the stopping criterion. Indeed, by the initial definition of I n , τ = inf {n ∈ N : (n + 1)b(n, α, S n ) ≤ } , which is computationally easier to check.
For comparisons with SIMCTEST, it will be useful to write τ equivalently as τ = inf{n ∈ N : S n ≥ u n or S n ≤ l n }, where u n = max{k : (n + 1)b(n, α, k) > } + 1 and l n = min{k : (n + 1)b(n, α, k) > } − 1 for any n ∈ N. We call (l n ) n∈N and (u n ) n∈N the (implied) stopping boundaries. Figure 1 illustrates the implied stopping boundaries for different testing thresholds α.
In order to define the resampling risk for CSM we define an estimatorp c of p aŝ The following theorem shows that the resampling risk of CSM is uniformly bounded. The proof of Theorem 1 can be found in Appendix A.
Will CSM stop in a finite number of steps? If p = α then the algorithm will only stop with probability of at most . Indeed, P α (τ < ∞) = P α (∃n ∈ N : α / ∈ I n ) ≤ by construction. However, if p = α then the algorithm will stop in a finite number of steps with probability one. Lai (1976) shows that with probability one, lim n→∞ f n (S n ) = p = lim n→∞ g n (S n ) given p = α, thus implying the existence of n ∈ N such that α / ∈ I n . Figure 2 shows the expected number of steps as a function of p for three different values ∈ {0.01, 0.001, 0.0001}. The testing threshold chosen for Figure 2 was α = 0.05. In Figure 2, the expected number of steps increases slightly as the resampling risk decreases. For a fixed value of the resampling risk, the effort increases when the true p-value approaches the threshold α.
If one considers a setup in which p is random (e.g., for power/level computations or in a Bayesian setup), then E[τ ] = ∞. This is a consequence of (Wald, 1945, Equation 4.81) which also applies to any procedure satisfying (1) for < 0.5 (see also (Gandy, 2009, Section 3.1)). To have a feasible algorithm for a random p, a finite upper threshold on the number of steps of CSM has to be imposed. Alternatively, a specialised procedure can be used (e.g., Gandy and Rubin-Delanchy (2013)).

Review of SIMCTEST
This section reviews the SIMCTEST method of Gandy (2009) (Sequential Implementation of Monte Carlo Tests) which also bounds the resampling risk uniformly. SIMCTEST sequentially updates two integer sequences (L n ) n∈N and (U n ) n∈N serving as lower and upper stopping boundaries and stops the sampling process once the trajectory (n, S n ) hits either boundary. The decision whether the p-value lies above (below) the threshold depends on whether the upper (lower) boundary was hit first.
The boundaries (L n ) n∈N and (U n ) n∈N are a function of α, computed recursively such that the probability of hitting the upper (lower) boundary, given p ≤ α (p > α), is less than . Starting with U 1 = 2, L 1 = −1, the boundaries are recursively defined as where n , n ∈ N, is called a spending sequence. The spending sequence is non-decreasing and satisfies n → as n → ∞ as well as 0 ≤ n < for all n ∈ N. Its purpose is to control how the overall resampling risk is spent over all iterations of the algorithm: In any step n of SIMCTEST, new boundaries are computed with a risk of n − n−1 . Gandy (2009) suggests n = n/(n + k) as a default spending sequence, where k is a constant, and chooses k = 1000. SIMCTEST stops as soon as the trajectory (n, S n ) hits the lower or upper boundary, thus leading to the stopping time σ = inf{k ∈ N : S k ≥ U k or S k ≤ L k }. In this case, a p-value estimate can readily be computed asp s = S σ /σ if σ < ∞ andp s = α otherwise. Similarly to Figure 2, the expected stopping time of SIMCTEST diverges as p approaches the threshold α.
SIMCTEST achieves the desired uniform bound on the resampling risk under certain conditions. To be precise, Theorem 2 in Gandy (2009) states that if ≤ 1/4 and log( n − n−1 ) = o(n) as n → ∞, then (1) holds withp =p s .

Comparison of CSM to SIMCTEST with the default spending sequence
In this section we compare the asymptotic behaviour of the width of the stopping boundaries for CSM and SIMCTEST. SIMCTEST is employed with the default spending sequence given in Section 3. Unless stated otherwise, we always consider the threshold α = 0.05 and aim to control the resampling risk at = 10 −3 .

Comparison of boundaries
We start by comparing the stopping boundaries of CSM and SIMCTEST. First, Figure 3 gives an overview of the upper and lower stopping boundaries of CSM and SIMCTEST up to 5000 steps, respectively. Second, Figure 4 shows the ratio of the widths of the stopping boundaries for both methods, that is (u n − l n )/(U n − L n ), up to 10 7 steps, where u n , l n (U n , L n ) are the upper and lower stopping boundaries of CSM (SIMCTEST). According to Figure 4, the boundaries of CSM are initially tighter than the ones of SIM-CTEST but become wider as the number of steps increases. However, this will eventually reverse again for large numbers of steps as depicted in Figure 4.

Real resampling risk in CSM and SIMCTEST
Both SIMCTEST and CSM are guaranteed to bound the resampling risk by some constant chosen in advance by the user. We will demonstrate in this section that the actual resampling risk (that is the actual probability of hitting a boundary leading to a wrong decision in any run of an algorithm) for SIMCTEST is close to , whereas CSM does not make full use of the allocated resampling risk. This in turn indicates that it might be possible to construct boundaries for SIMCTEST which are uniformly tighter than the ones of CSM; we will pursue this in Section 5.
We compute the actual resampling risk recursively for both methods by calculating the probability of hitting the upper or the lower stopping boundary in any step for the case p = α; other values of p give smaller resampling risks. This can be done as follows: Suppose we know the distribution of S n−1 conditional on not stopping up to step n − 1. This allows us to compute the probability of stopping at step n as well as to work out the distribution of S n conditional on not stopping up to step n. Figure 5 plots the cumulative probability of hitting the upper and lower boundaries over 5 · 10 4 steps for both methods. As before we control the resampling risk at our default choice of = 10 −3 . SIMCTEST seems to spend the full resampling risk as the number of samples goes to infinity. Indeed, the total probabilities of hitting the upper and lower boundaries in SIMCTEST are both 9.804 · 10 −4 within the first 5 · 10 4 steps. This matches the allowed resampling risk up to that point of 50000 = (5 · 10 4 )/(5 · 10 4 + 1000) ≈ 9.804 · 10 −4 allocated by the spending sequence, which is close to the full resampling risk = 10 −3 .
CSM tends to be more conservative as it does not spend the full resampling risk. Indeed, the total probabilities of hitting the upper and lower boundaries in CSM up to step 5 · 10 4 are 4.726 · 10 −4 and 4.472 · 10 −5 , respectively. In particular, the probability of hitting the lower boundary in CSM is far less than .
This imbalance is more pronounced for even smaller thresholds. We repeated the computation depicted in Figure 5 for α ∈ {0.02, 0.01, 0.005} (figure not included), confirming that the total probabilities of hitting the upper and lower boundaries in CSM both decrease monotonically as α decreases.
5 Spending sequences which dominate CSM 5.1 Example of a bespoke spending sequence One advantage of SIMCTEST lies in the fact that it allows control over the resampling risk spent in each step through suitable adjustment of its spending sequence n , n ∈ N. This can be useful in practical situations in which the overall computational effort is limited. In such cases, SIMCTEST can be tuned to spend the full resampling risk over the maximum number of samples. On the contrary, CSM has no tuning parameters and hence does not offer a way to influence how the available resampling risk is spent.
Suppose we are given a lower bound L and an upper bound U for the minimal and maximal number of samples to be spent, respectively. We construct a new spending sequence in SIMCTEST which guarantees that no resampling risk is spent over both the first L samples as well as after U samples have been generated. We call this the truncated spending sequence: As expected, for the first 100 steps the stopping boundaries of SIMCTEST are much wider than the ones of CSM since no resampling risk is spent.
As soon as SIMCTEST starts spending resampling risk on the computation of its stopping boundaries, the upper boundary drops considerably. By construction, the truncated spending sequence is chosen in such a way as to make SIMCTEST spend all resampling risk within 10 4 steps. Indeed, we observe in Figure 6 that as expected, the stopping boundaries of SIMCTEST are uniformly narrower than those of CSM over the interval (L, U), thus resulting in a uniformly shorter stopping time for SIMCTEST.
We also observe, however, that this improvement in the width of the stopping boundaries seems to be rather marginal, making the tuning-free CSM method a simple and appealing competitor.

Uniformly dominating spending sequence
Section 5.1 gave an example in which it was possible to choose the spending sequence for SIMCTEST in such a way as to obtain stopping boundaries which are strictly contained within the ones of CSM for a pre-specified range of steps.
Motivated by Figure 5 indicating that CSM does not spend the full resampling risk, we aim to construct a spending sequence with the property that the resulting boundaries in SIMCTEST are strictly contained within the ones of CSM for every number of steps. This implies that the stopping time of SIMCTEST is never longer than the one of CSM. Our construction is dependant on the specific choice α = 0.05. We define a new spending sequence for SIMCTEST as n = n 0.5 /(n 0.5 + 3) · , n ∈ N. Appendix B shows that this sequence empirically matches the rate at which SIMCTEST spends the real resampling risk with the one of CSM. Figure 7 depicts the differences between the upper and lower boundaries of CSM (upper u n , lower l n ) and SIMCTEST (upper U n , lower L n ) with the aforementioned spending sequence. We observe that l n ≤ L n as well as u n ≥ U n for n ∈ {1, . . . , 5 · 10 4 }, thus demonstrating that SIMCTEST can be tuned empirically to spend the resampling risk at the same rate as CSM while providing strictly tighter upper and lower stopping boundaries (over a finite number of steps). We observe that the gap between the boundaries seems to increase with the number of steps, leading to the conjecture that SIMCTEST has tighter boundaries for all n ∈ N.

Comparison with other truncated sequential Monte Carlo procedures
In this section, we compute the resampling risk for several truncated procedures as a function of p and thus demonstrate that they do not bound the resampling risk uniformly. We consider truncated versions of CSM and SIMCTEST, which we denote by tCSM and tSIMCTEST, as well as the algorithms in Besag and Clifford (1991); Davidson and MacKinnon (2000); Fay et al. (2007); Silva and Assunção (2018). The maximum number of samples is set to 13000 to roughly fit in with the maximum number of steps of Davidson and MacKinnon (2000). The methods of Besag and Clifford (1991); Davidson and MacKinnon (2000); Fay et al. (2007); Silva and Assunção (2018) have tuning parameters, which we choose as follows. As suggested in Besag and Clifford (1991), the recommended parameter h controlling the number of exceedances is set to 10. For Davidson and MacKinnon (2000), the pretest level β concerning the level and power of the test is set to 0.05, and the minimum and maximum  Figure 8 shows the resampling risk as a function of the p-value. As expected, a truncated test results in a resampling risk of at least 50% when p = α. For other p-values than p = α, the resampling risk can be smaller depending on the type of algorithm used and the number of samples it draws. However, tCSM is still amongst the best performers as it yields a low resampling risk almost everywhere with a localised spike at 0.5. It also guarantees to bound the risk uniformly when the number of simulations tends to infinity while other truncated procedures Besag and Clifford (1991); Davidson and MacKinnon (2000); Fay et al. (2007) do not have this property.
Ruxton and Neuhäuser Ruxton and Neuhäuser (2013) employ SIMCTEST to conduct a hypothesis test to determine whether the means of the penguin counts on Stewart Island are equal to the ones of the cat-free islands. They apply Welch's t-test (Welch, 1947) to assess whether two population groups have equal means. The test statistic of Welch's t-test is given as where n 1 , n 2 are the sample sizes,μ 1 ,μ 2 are the sample means and s 2 1 , s 2 2 are the sample variances of the two groups.
Under the assumption of normality of the two population groups, the distribution of the test statistic T under the null hypothesis is approximately a Student's t-distribution with v degrees of freedom, where .
Using the above data, we obtain t = −0.45 as the observed test statistic and a p-value of 0.09. As the normality assumption may not be satisfied in our case and as the t-distribution is only an approximation, Ruxton and Neuhäuser Ruxton and Neuhäuser (2013) implement a parametric bootstrap test which randomly allocates each of the 178 penguin pairs to one of the 29 islands, where each island is chosen with equal probability. Based on their experiments, they conclude that they cannot reject the null hypothesis at a 5% level.
Likewise, we apply CSM and SIMCTEST with the same bootstrap sampling procedure. We record the average effort measured in terms of the total number of samples generated. We set the resampling risk to = 0.001 and use SIMCTEST with its default spending sequence n = n/(n + 1000).
We first perform a single run of both CSM and SIMCTEST. CSM and SIMCTEST stop after 751 and 724 steps with p-value estimates of 0.09 and 0.08, respectively. Hence, both algorithms reject the null hypothesis. We then conduct 10000 independent runs to stabilise the results. Amongst those 10000 runs, CSM rejects the null hypothesis 10000 times compared with 9999 times for SIMCTEST. The average efforts of CSM and SIMCTEST are 1440 and 1131, respectively. Therefore, in this example, CSM gives comparable results to SIMCTEST while generating more samples on average. We expect such behaviour due to the wider stopping boundaries of CSM in comparison with SIMCTEST (see Figure 3). However, we need to pre-compute the stopping boundaries of SIMCTEST in advance, which is not necessary in CSM.

SIMCTEST
SIMCTEST with precomputed boundaries Memory requirement { n } n∈N Implementation from scratch Very easy Easy Easy Table 1: Comparison between CSM and SIMCTEST. The parameter τ denotes the stopping time and τ max denotes the maximum length of the pre-computed boundaries for SIMCTEST. Empirical quantities are denoted with * .

Discussion
This article introduces a new method called CSM to decide whether an unknown p-value, which can only be approximated via Monte Carlo sampling, lies above or below a fixed threshold α while uniformly bounding the resampling risk at a user-specified > 0. The method is straightforward to implement and relies on the construction of a confidence sequence (Robbins, 1970;Lai, 1976) for the unknown p-value. We compare CSM to SIMCTEST (Gandy, 2009), finding that CSM is the more conservative method: The (implied) stopping boundaries of CSM are generally wider than the ones of SIMCTEST and in contrast to SIMCTEST, CSM does not fully spend the allocated resampling risk .
We use these findings in two ways: First, an upper bound is usually known for the maximal number of samples which can be spent in practical applications. We construct a truncated spending sequence for SIMCTEST which spends all the available resampling risk within a prespecified interval, thus leading to uniformly tighter stopping boundaries and shorter stopping times than CSM. Second, we empirically analyse at which rate CSM spends the resampling risk. By matching this rate with a suitably chosen spending sequence, we empirically tune the stopping boundaries of SIMCTEST to uniformly dominate those of CSM even for open-ended sampling.
A comparison of memory requirement and computational effort for CSM and SIMCTEST is given in Table 8. In SIMCTEST, the boundaries are sequentially calculated as further samples are being generated whereas in SIMCTEST with pre-computed boundaries, the boundaries are initially computed and stored up to a maximum number of steps τ max . In CSM, solely the cumulative sum S n needs to be stored in each step, leading to a memory requirement of O(1). Gandy (2009) empirically shows that SIMCTEST with the default spending sequence has a memory requirement of O( √ τ log τ ). In SIMCTEST with pre-computed boundaries and default spending sequence, the amount of memory required temporarily up to step n is O( √ n log n). To compute the boundaries up to τ max , a total memory of O( √ τ max log τ max ) is hence required. Additionally, the values of the upper and lower boundaries up to τ max need to be stored, which requires O(τ max ) memory. Hence, the total memory requirement of SIM-CTEST with pre-computed boundaries is O(τ max ). Evaluating the stopping criterion in each step of CSM or SIMCTEST with pre-computed boundaries requires O(1), leading to the total computational effort of O(τ ) depicted in Table 8 for both cases. The computational effort of SIMCTEST is roughly proportional to τ n=1 |U n − L n | (Gandy, 2009). Using the empirical result |U n − L n | ∼ O( √ n log n), we obtain a bound of O(τ √ τ log τ ) for the computational effort of SIMCTEST.
We also compare the truncated versions of CSM (tCSM) and SIMCTEST (tSIMCTEST) with other truncated sequential Monte Carlo procedures. We demonstrate empirically that the resampling risk of the truncated methods cannot be bounded by an arbitrary small number and exceeds 0.5 when the true p-value equals the threshold. Nevertheless, tCSM and tSIMCTEST are still among the best performers, meaning that they yield a low resampling risk almost everywhere with a localised spike at 0.5.
The advantage of SIMCTEST (with pre-computed boundaries) lies in its adjustable spending sequence { n } n∈N : This flexibility allows the user to control the resampling risk spent in each step, thus enabling the user to spend no risk before a pre-specified step or to spend the full risk within a finite number of steps (see Section 5.1). This leads to (marginally) tighter stopping boundaries and faster decisions. The strength of CSM, however, lies in its straightforward implementation compared to SIMCTEST. Both methods illustrate a superior performance (measured in resampling risk) if truncations are applied. Overall we conclude that the simplicity of CSM, and its comparable performance to SIMCTEST and other truncated procedures make it a very appealing competitor for practical applications.
A Proof of Theorem 1 Proof. We start by considering the case p ≤ α. If p ≤ α, the resampling risk is RR p (p c ) = P p (p c > α). We show thatp c > α only when hitting the upper boundary and that the probability of hitting the upper boundary is bounded by .
To see the former: When not hitting any boundary, i.e. on the event {τ = ∞}, we havê p c = α. When hitting the lower boundary, i.e. on the event {τ < ∞, S τ ≤ l τ }, we havê p c = S τ /τ ≤ l τ /τ . It thus suffices to show l n /n ≤ α for all n ∈ N.
To finish the proof for this case we show that the probability of hitting the upper boundary is bounded by , which can be done using (Gandy, 2009, Lemma 3) and (2): The case p > α can be proven analogously to the case p < α using that P p (τ = ∞) = 0, which is shown in (Lai, 1976, p. 268).

B Finding a uniformly dominating spending sequence
In Section 5.2 we aim to find a spending sequence in SIMCTEST so that its boundaries are strictly contained within the ones of CSM for every number of steps.
We achieve this by first determining the (empirical) rate at which the real resampling risk is spent in each step in CSM. By matching this rate using a suitably chosen spending sequence, we obtain upper and lower stopping boundaries for SIMCTEST which are uniformly narrower than the ones of CSM (verified for the first 5 · 10 4 steps).
We start by estimating the rate at which the real resampling risk is spent in CSM. We are interested in empirically finding an l ∈ R such that n l · CSM n − CSM n−1 is constant, where CSM n is the (cumulative) real resampling risk (the total probability of hitting either boundary) for the first n steps in CSM. Figure 9 depicts n l · CSM n − CSM n−1 for both the upper (left plot) and lower (right plot) boundary of CSM as a function of the number of steps n and for three values l ∈ {1.4, 1.5, 1.6}. Based on Figure 9 we estimate that CSM spends the resampling risk at roughly O(n −1.5 ) for both the upper and lower boundaries.
We start with an analytical calculation of the rate at which SIMCTEST spends the resampling risk (as opposed to also estimating it). The default spending sequence in SIM-CTEST is S n = n/(n + k) , n ∈ N. Hence the resampling risk spent in step n ∈ N is ∆ S n = S n − S n−1 = k/((n + k)(n + k − 1)) ∼ n −2 . We conducted simulations (similar to the ones in Figure 9 for CSM) which indeed confirm the analytical O(n −2 ) rate for SIMCTEST (simulations not included in this article). Overall, the spending rate of O(n −1.5 ) for CSM is thus slower than the O(n −2 ) rate of SIMCTEST with the default spending sequence.
In order to match the O(n −1.5 ) rate for CSM, we generalise the default spending sequence of SIMCTEST to S n = n γ / (n γ + k) for n ∈ N and a fixed γ > 0. Similarly to the aforementioned derivation, SIMCTEST in connection with S n will spend the real resampling risk at a rate of O n −(γ+1) . We choose the parameters γ and k to obtain stopping boundaries for SIMCTEST which dominate the ones of CSM. First, we set γ = 0.5 to match the rate of CSM. Second, we empirically determine k to keep the stopping boundaries of SIMCTEST within the ones of CSM (for the range of steps n ∈ {1, . . . , 5 · 10 4 } considered in Figure 7). We find that the choice k = 3 satisfies this condition.