Risk-Limiting Audits by Stratified Union-Intersection Tests of Elections (SUITE)

  • Kellie OttoboniEmail author
  • Philip B. StarkEmail author
  • Mark Lindeman
  • Neal McBurnett
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11143)


Risk-limiting audits (RLAs) offer a statistical guarantee: if a full manual tally of the paper ballots would show that the reported election outcome is wrong, an RLA has a known minimum chance of leading to a full manual tally. RLAs generally rely on random samples. Stratified sampling—partitioning the population of ballots into disjoint strata and sampling independently from the strata—may simplify logistics or increase efficiency compared to simpler sampling designs, but makes risk calculations harder. We present SUITE, a new method for conducting RLAs using stratified samples. SUITE considers all possible partitions of outcome-changing error across strata. For each partition, it combines P-values from stratum-level tests into a combined P-value; there is no restriction on the tests used in different strata. SUITE maximizes the combined P-value over all partitions of outcome-changing error. The audit can stop if that maximum is less than the risk limit. Voting systems in some Colorado counties (comprising 98.2% of voters) allow auditors to check how the system interpreted each ballot, which allows ballot-level comparison RLAs. Other counties use ballot polling, which is less efficient. Extant approaches to conducting an RLA of a statewide contest would require major changes to Colorado’s procedures and software, or would sacrifice the efficiency of ballot-level comparison. SUITE does not. It divides ballots into two strata: those cast in counties that can conduct ballot-level comparisons, and the rest. Stratum-level P-values are found by methods derived here. The resulting audit is substantially more efficient than statewide ballot polling. SUITE is useful in any state with a mix of voting systems or that uses stratified sampling for other reasons. We provide an open-source reference implementation and exemplar calculations in Jupyter notebooks.


Stratified sampling Nonparametric tests Fisher’s combining function Sequential hypothesis tests Colorado risk-limiting audits Maximizing P-values over nuisance parameters Union-intersection test Intersection-union test 



We are grateful to Ronald L. Rivest and Steven N. Evans for helpful conversations and suggestions.


  1. 1.
    Bañuelos, J., Stark, P.: Limiting risk by turning manifest phantoms into evil zombies. Technical report, (2012). Accessed 17 July 2012
  2. 2.
    California Secretary of State: California Secretary of State Post-Election Risk-Limiting Audit Pilot Program 2011–2013: Final Report to the United States Election Assistance Commission (2014). Accessed 6 May 2018
  3. 3.
    Grimmett, G.R., Stirzaker, D.R.: Probability and Random Processes. Oxford University Press, Oxford, August 2001.
  4. 4.
    Higgins, M., Rivest, R., Stark, P.: Sharper p-values for stratified post-election audits. Stat. Polit. Policy 2(1) (2011).
  5. 5.
    Lindeman, M., Stark, P., Yates, V.: BRAVO: ballot-polling risk-limiting audits to verify outcomes. In: Proceedings of the 2011 Electronic Voting Technology Workshop/Workshop on Trustworthy Elections (EVT/WOTE 2011). USENIX (2012)Google Scholar
  6. 6.
    Lindeman, M., Stark, P.B.: A gentle introduction to risk-limiting audits. IEEE Secur. Priv. 10, 42–49 (2012)CrossRefGoogle Scholar
  7. 7.
    Pesarin, F., Salmaso, L.: Permutation Tests for Complex Data: Theory, Applications, and Software. Wiley, West Sussex (2010)CrossRefGoogle Scholar
  8. 8.
    Rivest, R.L.: Bayesian tabulation audits: explained and extended, 1 January 2018.
  9. 9.
    Stark, P.: Conservative statistical post-election audits. Ann. Appl. Stat. 2, 550–581 (2008).
  10. 10.
    Stark, P.: Auditing a collection of races simultaneously. Technical report. (2009).
  11. 11.
    Stark, P.: CAST: canvass audits by sampling and testing. IEEE Trans. Inf. Forensics Secur. Spec. Issue Electron. Voting 4, 708–717 (2009)CrossRefGoogle Scholar
  12. 12.
    Stark, P.: Risk-limiting post-election audits: \(P\)-values from common probability inequalities. IEEE Trans. Inf. Forensics Secur. 4, 1005–1014 (2009)CrossRefGoogle Scholar
  13. 13.
    Stark, P.: Risk-limiting vote-tabulation audits: the importance of cluster size. Chance 23(3), 9–12 (2010)CrossRefGoogle Scholar
  14. 14.
    Stark, P.: Super-simple simultaneous single-ballot risk-limiting audits. In: Proceedings of the 2010 Electronic Voting Technology Workshop/Workshop on Trustworthy Elections (EVT/WOTE 2010). USENIX (2010).
  15. 15.
    Stark, P.B., Teague, V.: Verifiable European elections: risk-limiting audits for D’Hondt and its relatives. JETS: USENIX J. Election Technol. Syst. 3(1) (2014).
  16. 16.
    Stark, P.B., Wagner, D.A.: Evidence-based elections. IEEE Secur. Priv. 10, 33–41 (2012)CrossRefGoogle Scholar
  17. 17.
    Wald, A.: Sequential tests of statistical hypotheses. Ann. Math. Stat. 16, 117–186 (1945)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.Department of StatisticsUniversity of CaliforniaBerkeleyUSA
  2. 2.Verified Voting FoundationPhiladelphiaUSA
  3. 3.BoulderUSA

Personalised recommendations