The Value of the Election Administration and Voting Survey

Abstract

This case study examines the implementation of the Election Administration and Voting Survey (EAVS). EAVS is the only national survey to capture data related to the voting and elections process in every state and generates a phenomenal amount of data. Of course, the numbers most people care about are the results—how many votes the candidate they support receives. But beyond the results is a wealth of data related to how elections are run, from how many citizens are registered to vote to how many people cast ballots during early voting, to the number of provisional ballots cast, counted, and rejected. The EAVS captures this data and tells the story of the nuts and bolts of the voting process for both the voters and the election officials charged with administering elections. The importance of this type of data collection, what goes into administering a survey of this scope, the challenges in collecting this vast amount of data, and the potential for improvement will be discussed.

Keywords

Elections generate a phenomenal amount of data. Of course, the numbers most people care about are the results—how many votes the candidate they support receives. But beyond the results is a wealth of data related to how elections are run, from how many citizens are registered to vote to how many people cast ballots during early voting, to the number of provisional ballots cast, counted, and rejected. These data tell the story of the nuts and bolts of the voting process for both the voters and the election officials charged with administering elections.

From June 2016 to June 2018 as an elections specialist and director of research at the US Election Assistance Commission (EAC) I managed the only national survey to capture this type of information for all states, the Election Administration and Voting Survey (EAVS). In this case study I will discuss the importance of this type of data collection, what goes into administering a survey of this scope, the challenges in collecting this vast amount of data, and the potential for improvement.

What Is the EAVS and Why Is It Important?

The EAVS is a biennial survey which collects, analyzes, and reports on state-by-state data related to the administration of federal elections. It has been administered since 2004 and includes data about voter registration, military and overseas voters, mail and absentee voting, poll workers and polling places, provisional ballots, and voting technology. An additional survey initially called the Statutory Overview (now called the Policy Survey), first administered in 2008, provides information about state law and practices to inform and provide context to data reported in the EAVS.

Why is this important? This data and analysis can provide voters, elected officials, and the media a deeper understanding of how elections are run as well as identify trends and emerging challenges. For election officials, it can provide information not only about their own jurisdiction, but about neighboring jurisdictions and insight into best practices they or other jurisdictions may be using. Using and applying lessons from data analysis can lead to new and innovative ways to manage elections.

The Nuts and Bolts of Analyzing the Nuts and Bolts

The EAVS is an enormous endeavor, both for those administering it and for those responding to it. It asks hundreds of questions and generates hundreds of thousands of data points based on local jurisdiction-level data from all 50 states, the District of Columbia, American Samoa, Guam, Puerto Rico, and the US Virgin Islands.

In this section I will provide a picture, albeit incomplete, of some of what goes into implementing the EAVS. I will focus on five areas:

• Survey approval and public comment;

• Who administers the EAVS;

• Technical assistance;

• Who responds to the EAVS; and

• Completing the survey and validating the data.

Survey Approval

Through the Paperwork Reduction Act (PRA) the federal government requires that information collections like the EAVS be approved by the Office of Management and Budget. This is to ensure that undue burdens are not placed on the public—in the case of the EAVS the states—in responding to these types of requests from the federal government.

Although there is a touch of irony that the PRA involves completing some paperwork (electronically at least), it is an important reminder that asking states to respond to a complex and time-consuming survey is not to be undertaken lightly. Additionally, during this review process which can take four to six months, there are two mandated and incredibly helpful public comment periods. The survey questions are published and in 2016 the EAC received dozens of comments from concerned individuals, advocacy groups, as well as election officials.

Who Administers EAVS

The EAC administers the EAVS through a contractor hired via a competitive bid process. This has been the case for every EAVS since its inception in 2004. The contractor is the entity that sends out the survey to states, assists jurisdictions during the data collection process, provides analysis, and drafts reports. As I will discuss below, working with a contractor can provide a number of advantages including allowing the EAC to work with some of the foremost experts in the field as well as those who are experts in survey administration.

Technical Assistance

Another advantage to working with a contractor is the staff, time, and resources that a contractor can utilize. During my time at the EAC there was a staff of fewer than 30 people and of those only myself and one other staff member was focused on research and the EAVS. The EAC has many other responsibilities and manages to accomplish a great deal with a lean staff. However, two researchers are not enough to administer, manage, and provide all the assistance the states need when responding to a survey of this magnitude.

One of the key components of administering this survey is providing technical assistance to states at all stages of the survey. For the 2016 EAVS two examples of this assistance come to mind that were critical in successfully conducting the survey. First was an in-depth needs assessment of each state in the summer before the November election. This allowed the contractor to establish a working relationship with the appropriate points of contact and get specific information from the states on how they would respond to the survey and any particular needs or limitations they had.

Second, after the election when states were in the midst of responding to the survey the contractor had nearly ten technical assistants tasked to help states answer any questions they had about the survey. They responded to hundreds of inquiries and were able to assist states on a variety of issues, from technical questions about completing the spreadsheet, to larger questions about definition of terms in the survey.

Who Responds to the EAVS

The simple answer is that the states, the District of Columbia, American Samoa, Guam, Puerto Rico, and the US Virgin Islands respond to the survey. In reality, this varies a great deal from state to state. The EAVS gathers data from states at the jurisdictional level. In most states this is at the county level, and some is at the city or township level. States are responsible for collecting this data for all their jurisdictions.

Several states in 2016 were able to respond to a majority of the survey using information in their statewide election management systems—sometimes these are referred to as top-down states, where the information is gathered and to some extent controlled at the state level. These states often have much of the EAVS questions pre-programmed into their statewide management systems and can generate responses with relative ease. One challenge for top-down states is ensuring the data generated for the EAVS from their systems can be converted into the EAVS template for submission.

On the other end of the spectrum are bottom-up states where the survey questions need to be sent to each jurisdiction and then each jurisdiction needs to respond and send the survey back to the state. The states then need to combine all these jurisdictions’ data into the EAVS format and submit it in one file. This can present numerous challenges. In some cases states have little authority to ensure jurisdictions respond. And for jurisdictions not familiar with the survey it can be confusing. In these cases, the local jurisdictions often reach out for technical assistance. And in several jurisdictions, responses to the 2016 survey were completed by hand and sent by mail.

Completing and Validating the Survey

States respond to the EAVS using an Excel template. This Excel document includes a series of macros to help the respondent enter and review the data for errors. In 2016 another Excel template was created to allow for copying and pasting jurisdiction-level data. However, Excel is not the ideal mode to collect this amount of data from these many jurisdictions—more on that when I discuss potential improvements to the EAVS.

After states submit the data, they are reviewed for accuracy. This includes attempting to determine what any empty cells represent—data that are missing, data left blank because it is not applicable for a state (i.e. the state does not have Election Day registration and therefore has no data for this question), or a true value of zero. Or was it mistakenly left empty? Attempts are made to catch math errors as well. An example is finding an impossible value, such as a state showing it had more rejected provisional ballots than provisional ballots issued. And in 2016 for the first time more advanced statistical methods were used to see if the data fell between what would be expected of a jurisdiction based on data from previous years and the characteristics of the jurisdiction. Reports were generated and sent to each state flagging possibly problematic data and providing an opportunity for states to make changes or confirm the data is accurate.

All Challenges, Great and Small

As the previous section demonstrates, administering the EAVS is not a small task and the devil is most definitely in the details. Pulling back for a bigger-picture view, one sees these details inform the high-level challenges and opportunities for the EAVS as well.

One challenge for the field of election administration which has an impact on the EAVS is terminology and the lack of accepted common terms across the states. For example, the phrase “early voting” may mean in-person voting on a voting machine during a set time before Election Day in some states, while in others it may be casting an absentee ballot in person at an election office before Election Day. Yet, in other states, in-person absentee voting is considered a part of absentee voting. While the EAVS attempts to provide clear instructions and definitions within the survey instrument, respondents will of course bring their own definitions to the table.

Another challenge is whether the right questions are being asked and being asked in the best way possible. The survey has been essentially the same since 2008 and changing the survey cannot be done easily or taken lightly. First, changes require the federal review process noted above that must be followed any time changes to the survey are made. Second and more importantly, election officials need to be given a good deal of lead time to deal with any significant changes in order to make changes in the methods that they use to collect the data. And they need to be involved, along with survey experts, when these changes are made.

That said, there are times for review and change. In the past, a working group was created to review Section B of the survey about military and overseas voters, mostly to streamline the questions in that section. And this type of review could certainly be helpful for other sections, including Section A related to voter registration.

Another area where there is room to improve is the use of the Statutory Overview/Policy Survey. In the past it has been released separately from the EAVS. However, it provides important context to the data in the EAVS. For example, knowing about why states issue and reject provisional ballots goes a long way in understanding the variation in these data across states. In the future linking these surveys more directly or even combining them could add even more power to the EAVS data and analysis.

This leads to the last but possibly most important potential change for the EAVS—shifting from an Excel-based template to an online survey tool. The introduction of an online data collection system could lead to more accurate data; reduce the burden and simplify the process for respondents; provide real-time error checks during data collection; allow all states but especially bottom-up states a more manageable way to share the survey at the local level; and allow for the incorporation of the Statutory Overview/Policy Survey into the EAVS, building in related legal and process questions into the appropriate sections.

Conclusion

While the EAVS faces some challenges, it is an amazing and unique source of election administration data. And every year it is issued, the survey gains power, with more data over more elections allowing for better comparisons within and between states and jurisdictions, as well as more opportunity to observe trends.

From experience I know a great deal of work goes into both managing the survey and responding to the survey. Understanding and answering all the questions, verifying the data, and providing the appropriate analysis is labor intensive and difficult. However, as the survey has already demonstrated and with these potential improvements, it is and can continue to be an invaluable resource for all those who care about the nuts and bolts of our democracy.