Back
Thursday | 25 August, 2016

Making Votes Count: Statistical Anomalies in Election Statistics

[Note: In order to help monitor the fidelity of the October 2016 parliamentary election results, CRRC-Georgia will carry out quantitative analysis of election-related statistics using methods from the field of election forensics within the auspices of the Detecting Election Fraud through Data Analysis (DEFDA) project. The Project is funded by the Embassy of the United States of America in Georgia, however, none of the views expressed in the following blog post represent the views of the US Embassy in Georgia or any related US Government entity.]


On Friday, August 19th, CRRC-Georgia presented and published a pre-analysis report for the Detecting Election Fraud through Data Analysis project, which contained analysis of the new electoral boundaries set up following the 2015 constitutional court ruling that the previous boundaries were unconstitutional.


The report also demonstrated how the methods of statistical analysis that CRRC-Georgia will use to monitor the 2016 elections work in practice. To do so, we used precinct level data from the 2012 party list elections. Specifically, CRRC-Georgia carried out two types of statistical analyses:


  • Logical checks of official election returns, which test whether there were data entry errors when the vote was being recorded and collated;

  • Tests for statistical anomalies in the official electoral returns, which may suggest electoral malfeasance.

While Monday’s blog shows the logical checks that CRRC will apply to the final CEC vote records, today we discuss the tests used to identify statistical anomalies in vote counts.


Election Forensics: Detecting statistical anomalies in voting data


Direct observation of polling stations is the best method available to ensure the accuracy of the vote, however, election observers cannot be everywhere all the time. Given this fact, the field of election forensics, a subfield of political science, has developed a number of statistical tests to look for statistical anomalies in election returns, which may suggest suspicious election-related activity. Although a number of rather complicated statistics exist, we focus on a number of simpler tests. Specifically, we use tests based on the distribution of the second digit in the number of votes cast, the final digit in the number of votes cast, and the distribution of turnout within an electoral district.


Second digit tests are based on Benford’s law. Benford’s law provides the expected probability of the first digit being any digit one through nine in a number with multiple digits. Although one might expect this number to be equally likely to be any number, in fact 1 is more likely than 2, 2 more likely than 3, etc. Using Benford’s Law, accountants test various documents for anomalies that may suggest issues in documents. This law also applies to the second digit in a number, which researchers have found is more suitable for testing election results. A similar logic is applied to elections as in accounting, and in this blog, we specifically test whether the skew, kurtosis, and the average of the second digit and its distribution follow the expected distribution or not. Instances of non-conformity to Benford’s law may suggest electoral malfeasance.


Besides second digit tests, a number of tests have been proposed for the last digit in vote counts. Here, the expected distribution of digits is much more intuitive, and one expects each digit, zero through nine, to be approximately 10% of the total distribution. Based on this distribution, we test the mean of the last digit and of the mean of the count of zeros and fives in the final digits of votes.


In order to test whether the above noted digit tests in fact indicate potential issues or whether the difference between the observed and expected values was a chance variation, we use a statistical method called bootstrapping. This method lets us to estimate 99% confidence intervals. In the present case, the confidence intervals provide a range within which the result could have fallen by chance. If the range covered does not include the expected value for a given test statistic, we conclude with 99% confidence that the number is different not by chance alone.


Finally, voter turnout is expected to have a relatively normal distribution with a single mode. Based on this expectation, we test whether voter turnout in each electoral district has a single mode or multiple modes using what statisticians refer to as a dip test.


Before reporting the test results, it is worth noting several important caveats when interpreting these tests:


  • Test results are probabilistic, which means that they say the distribution is highly unlikely (would occur 1% of the time in the present case), rather than impossible to occur in the absence of issues. For the tests, we calculated 99% confidence intervals. With 99% confidence intervals and having conducted 444 tests on the 2012 proportional election results, statistically we would expect between four and five tests to be set off in the absence of issues due to chance alone.

  • The lack of a test being set off does not necessarily mean a problem occurred, but it does suggest the need for further examination;


In total, 11 districts show statistical anomalies in the test results, and a total of 15 tests report suspicious results. Results are presented in Table 5. In the rows with district names and numbers, the actual test values are reported. In the row below the district name, 99% confidence intervals are reported. Red cells in the table indicate the presence of a statistical anomaly.




 


Rustavi’s electoral returns set off three statistical tests. Given that we have no reason to expect specific voting patterns in Rustavi compared to other areas in the country that did not set off suspicious tests, this suggests that there may have been electoral malfeasance in Rustavi in 2012. Reviews of election monitoring reports, however, did not suggest electoral malfeasance. This test may be picking up on undetected electoral malfeasance from 2012 in Rustavi. Although unlikely, these three tests could have also been set off by chance.

 

In Kobuleti, two tests were also set off. In Kobuleti, we would not expect a particularly distinctive voting pattern. Hence, there is a relatively strong reason to believe that electoral malfeasance may have occurred in Kobuleti in the 2012 elections. This contention is supported by election monitoring reports, which reported issues in Kobuleti.

 

In Bolnisi, two tests were set off. Complaints were filed in Bolnisi on election day, and the test may have been set off by these issues. However, given Bolnisi’s relatively high ethnic minority population and distinctive voting pattern, the tests could have been set off by this rather than malfeasance.

 

Eight other districts had single positive tests for electoral malfeasance, including Vake, Saburtalo, Kareli, Akhaltsikhe, Adigeni, Vani, Senaki, and Martvili. A review of the OSCE and GYLA election monitoring reports suggest that issues may have occurred in at least half of these districts. Although these positive tests could have occurred by chance alone, the four districts in which a test was set off and observers did not report malfeasance in may also suggest unreported problems in the 2012 elections.

 

This blog post has described the methods CRRC-Georgia will use to detect statistical anomalies in election returns. For more on the methods CRRC-Georgia will use to monitor the elections, see our pre-analysis report, here, and take a look at Monday’s blog post on logical inconsistencies in election records.

22.02.2013 | Friday

Before and After the Elections: Shifting Public Opinion in Georgia

The Georgian parliamentary elections in October 2012 attracted much international interest and ushered in an important turn in Georgian politics. In 2012 CRRC conducted four waves of a Survey on Political Attitudes in Georgia for the National Democratic Institute (NDI) (funded by the Swedish International development Cooperation Agency-SIDA) in order to track changes in public opinion associated with these major political events.
23.06.2014 | Monday

Trust in local government in Georgia

On June 15th Georgian voters headed to the polls in local elections. There were problems leading up to the elections as detailed in last week's electoral notes. At present, results show a significant portion of positions in local government going to Georgian Dream Coalition (GD) candidates, though a number of races will go into second rounds
27.07.2011 | Wednesday

Rule of Law in Georgia - Opinions and Attitudes of the Population

As a part of the Caucasus Barometer Report Writing Competition held by CRRC in the spring of 2011, we would like to present the second report (the first report was published recently) written by Salome Tsereteli-Stephen. The report deals with the rule of law in Georgia and here is a short summary of Salome’s findings and an analysis of the subject.
03.05.2008 | Saturday

Exit Polls | Take Two

Readers may recall that we voiced some concern with regards to exit polls. Here is a fascinating account, first-hand, by a reputed pollster having what they describe as an "Adventure in Baku".
21.05.2008 | Wednesday

Parliamentary Elections in Georgia | ODIHR Observation

With today's elections in Georgia, various themes come to mind. Certainly, elections have come a long way: by now, the Georgian government employs a series of highly qualified consultants, including Greenberg Quinlan Rosner of Clinton-fame, plus a Brussels-based PR firm, as well as working with experienced teams from the Baltics. This, then, is no longer the game of the 1990s, or 2003. Election observers know that they in turn will be observed, and maybe that's how it should be.
23.10.2008 | Thursday

McCain vs Obama: Caucasus preferences


So here's something that we are a little puzzled about. The Economist is undertaking a poll to see which American Presidential candidate is favored by the world. In a very blue worldwide map, rooting for Obama, two noticeable yellowish spots, Macedonia and Georgia. McCain, of course, is popular in Georgia for having said "Today we all are Georgians" during the recent conflict.