Risk-Limiting Audits by Stratified Union-Intersection Tests of Elections (SUITE)
Risk-limiting audits (RLAs) offer a statistical guarantee: if a full manual tally of the paper ballots would show that the reported election outcome is wrong, an RLA has a known minimum chance of leading to a full manual tally. RLAs generally rely on random samples. Stratified sampling—partitioning the population of ballots into disjoint strata and sampling independently from the strata—may simplify logistics or increase efficiency compared to simpler sampling designs, but makes risk calculations harder. We present SUITE, a new method for conducting RLAs using stratified samples. SUITE considers all possible partitions of outcome-changing error across strata. For each partition, it combines P-values from stratum-level tests into a combined P-value; there is no restriction on the tests used in different strata. SUITE maximizes the combined P-value over all partitions of outcome-changing error. The audit can stop if that maximum is less than the risk limit. Voting systems in some Colorado counties (comprising 98.2% of voters) allow auditors to check how the system interpreted each ballot, which allows ballot-level comparison RLAs. Other counties use ballot polling, which is less efficient. Extant approaches to conducting an RLA of a statewide contest would require major changes to Colorado’s procedures and software, or would sacrifice the efficiency of ballot-level comparison. SUITE does not. It divides ballots into two strata: those cast in counties that can conduct ballot-level comparisons, and the rest. Stratum-level P-values are found by methods derived here. The resulting audit is substantially more efficient than statewide ballot polling. SUITE is useful in any state with a mix of voting systems or that uses stratified sampling for other reasons. We provide an open-source reference implementation and exemplar calculations in Jupyter notebooks.
KeywordsStratified sampling Nonparametric tests Fisher’s combining function Sequential hypothesis tests Colorado risk-limiting audits Maximizing P-values over nuisance parameters Union-intersection test Intersection-union test
We are grateful to Ronald L. Rivest and Steven N. Evans for helpful conversations and suggestions.
- 2.California Secretary of State: California Secretary of State Post-Election Risk-Limiting Audit Pilot Program 2011–2013: Final Report to the United States Election Assistance Commission (2014). http://votingsystems.cdn.sos.ca.gov/oversight/risk-pilot/final-report-073014.pdf Accessed 6 May 2018
- 3.Grimmett, G.R., Stirzaker, D.R.: Probability and Random Processes. Oxford University Press, Oxford, August 2001. www.amazon.ca/exec/obidos/redirect?tag=citeulike09-20&path=ASIN/0198572220
- 4.Higgins, M., Rivest, R., Stark, P.: Sharper p-values for stratified post-election audits. Stat. Polit. Policy 2(1) (2011). http://www.bepress.com/spp/vol2/iss1/7
- 5.Lindeman, M., Stark, P., Yates, V.: BRAVO: ballot-polling risk-limiting audits to verify outcomes. In: Proceedings of the 2011 Electronic Voting Technology Workshop/Workshop on Trustworthy Elections (EVT/WOTE 2011). USENIX (2012)Google Scholar
- 8.Rivest, R.L.: Bayesian tabulation audits: explained and extended, 1 January 2018. https://arxiv.org/abs/1801.00528
- 9.Stark, P.: Conservative statistical post-election audits. Ann. Appl. Stat. 2, 550–581 (2008). http://arxiv.org/abs/0807.4005
- 14.Stark, P.: Super-simple simultaneous single-ballot risk-limiting audits. In: Proceedings of the 2010 Electronic Voting Technology Workshop/Workshop on Trustworthy Elections (EVT/WOTE 2010). USENIX (2010). http://www.usenix.org/events/evtwote10/tech/full_papers/Stark.pdf
- 15.Stark, P.B., Teague, V.: Verifiable European elections: risk-limiting audits for D’Hondt and its relatives. JETS: USENIX J. Election Technol. Syst. 3(1) (2014). https://www.usenix.org/jets/issues/0301/stark