In the aftermath of the 2012 American presidential election, we can now issue a report card for several opinion polling organizations. How have some reliable pollsters fared?
Statistics as the Basis of Report Cards for Pollsters
In May 2012, “How Statistics May Help Select a Reliable Pollster” introduced the way statistics can measure the reliability of the predictions based on opinion polls.
Let’s follow up on Gallup and Rasmussen, and introduce Nate Silver too.
A Report Card for Gallup
The 2012 election results have updated the first line of data in the spreadsheet.
President Obama won, with a 2.8% margin in the popular vote. The Gallup organization predicted that Obama would have a 1% margin over challenger Mitt Romney, so Gallup’s score is -1.8%. In other words, Gallup had predicted too many votes for the losing candidate.
According to Politico, the popular vote was 62,611,250 to 59,134,475. Readers are invited to do the arithmetic to calculate the percentages earned by Obama and Romney.
By the way, Gallup’s prediction included “3% no opinion & 1% other” to create a 49-to-48% margin for Obama. When they forced the “3% no opinion” to actually decide, their internal models predicted a 50-49% split. Either way, Gallup predicted that Obama would have a 1% margin of victory in the popular vote.
A statistician’s viewpoint would be that Gallup’s standard deviation and confidence values improved with their prediction for 2012. The standard deviation went down from 3.20 to 3.12; and the confidence changed to 1.37 from 1.44.
Note that Gallup’s error of -1.8% is just a bit more than half of one standard deviation. This organization made a quite accurate prediction for the US presidential race in 2012.
A Report Card for Rasmussen
The Rasmussen organization has fewer years of predictions under its belt. They were completely accurate in predicting the state-by-state results in the 2004 and 2008 presidential elections.
We have reported on Rasmussen’s predictions of state-by-state electoral college results, rather than the popular vote.
In 2012, Rasmussen predicted that Obama would win 237 electoral college votes to lead Romney with 206, although 95 votes were listed as a “toss-up” in states with races too close to call.
The election resulted in 206 electoral college votes for Romney; all 95 toss-up votes went to Obama.
This poses a puzzle for the statistician’s report card: how to grade Rasmussen for the toss-up predictions?
My solution is to penalize Rasmussen with 95/2 = 47.5 electoral college votes too few for Obama, and the negative -8.8% score as the percentage of the total electoral college votes.
Rasmussen’s standard deviation score changed sharply, from zero to 5.08. It’s a high value both due to the one poor score, and because they have only three data points in their history. The confidence value can now be calculated; it is undefined when the standard deviation is zero.
If a teacher were to write a note on Rasmussen’s report card, it might say “Does well overall and should not be discouraged by one slip. Work on increased risk-taking for the future.”
Click to Read Page Two: Nate Silver’s Report Card
© Copyright 2012 Mike DeHaan, All rights Reserved. Written For: Decoded Science
Leave a Reply