Tiny Modelby Around The CounterThere Are No Good Polling Models
Do you remember election night 2016? I do. The mayor’s staff were all standing around together in a bar with our drinks ready to toast to Hillary Rodham Clinton’s victory. We were assured that the Democrats would be victorious.
Wrong.
2016 was an outlier cycle. Outliers occur more often than you'd think. From 2002 through 2024, there were 12 federal election cycles. Arguably 6 of them were outlier cycles.
From the election archive
Three Outlier Election Cycles
What the polls showed
What happened
2004
Early election-night exit poll
The Exit-Poll Mirage
U.S. president / National popular vote
- Exit-poll estimate
- Kerry +3.0
- Final result
- Bush +2.5
Early exit-poll figures put Kerry ahead. Bush won the popular vote. An election-night snapshot was not the result.
Nov. 2, 2004. Gallup reports Kerry 51%, Bush 48%; no specific release time is given. These were exit polls, not pre-election surveys.
2014
Pre-election polling average
The Underestimated Wave
Iowa / U.S. Senate
- Final RCP average
- Ernst +2.3
- Final result
- Ernst +8.3
The average picked the right winner, but not the size of her victory. Ernst won by a much wider margin than the polls suggested.
Polls: Oct. 27-Nov. 3, 2014; final average archived Nov. 5. Not every poll missed equally: the final Des Moines Register poll had Ernst ahead by 7.
2016
Pre-election polling average
The Swing-State Miss
Wisconsin / Presidential popular vote
- Final RCP average
- Clinton +6.5
- Final result
- Trump +0.8
Wisconsin's polling average gave Clinton a clear lead. Trump narrowly won the state. This was a state-level miss, not the national popular vote.
Polls: Oct. 26-Nov. 2, 2016. The average includes third-party candidates; the result uses final, post-recount vote totals.
Tiny Model Episode 1: There Are No Good Polling Models
(And We Mean It!)
Every election cycle, the polling models show up (e.g., in the NYT, The Economist, 538 (RIP 🪦), Silver Bulletin, etc.) to produce a fresh crop of election predictions. And every election cycle, the public incorrectly understands this to mean that it is possible to predict elections.
It is NOT POSSIBLE to predict elections.
In some election cycles (like 2004, 2014, or 2016), the polls are simply incorrect. Because of this, 0 polling models get all of these cycles right. Not 538, not NYT, not The Economist. Not even during back-testing. And, “back-testing” just means “we looked at how our model did in previous election cycles.” You can think of this a little like burning the sauce and claiming the next morning that you knew how to cook that gravy all along. Uh huh, sure.
When you think about it, this makes sense. You are building a polling model to predict election outcomes, and the data is completely wrong! How can you make accurate predictions if the data is completely wrong? Think back to our burnt sauce analogy. If you accidentally tossed a cup of sugar into the pot instead of a teaspoon of salt, no amount of simmering or added ingredients is going to turn that into something palatable. Bad polls as your foundation cause the exact same reaction in an election model, ruining the whole batch no matter how pretty the final chart looks.
The Simplest Possible Polling Model
As this series progresses towards the November election, we’ll develop increasingly sophisticated uses of this data for election prediction and bring you along with us. But first, let’s start with the simplest possible model: the generic-ballot polling average. This model asks:
If the election were held today, would the public generally support Democrats or Republicans? e.g., would the electorate be D+2, or R+2, or what?
This kind of model does not answer questions like “how many House seats will Republicans win”, or “who will win the presidency”, or anything like that. In fact, it actually doesn’t technically predict anything at all! Notice that the question specifically asks “If the election were held today.”
And this question is especially important because what it does is tell politicians exactly how people feel today. And that today is doing a lot of heavy lifting. Many people read a poll and instantly think “the Democrats will win the election because of these poll results.” That’s simply not correct. If the election were happening this very instant, perhaps the Democrats would win. But it’s almost never happening on the day and time the poll is in the field, so even when you’re reading the polling results, you should know that those are out-of-date results and any election that’s upcoming will likely be different from what you are reading.
Of course, we will develop models that can answer all of these questions the generic ballot omits and more. But we wanted to start with something simple and build into something more complex. Our goal is to bring you along with us as we develop from something naive to something complex. Through this process, we hope that you learn and grow with us!
So, What Does Go Into a Polling Model?
The goal of a polling model is to predict an election outcome. To predict an outcome, we’ve collected a whole bunch of data. Mostly polls, but also some information about the polls themselves that will determine how much “weight” we should give any individual poll.
Example polls
Weights for Siena/NYT
In other words, a poll gets more “weight” if it comes from a pollster with a long track record of polls that ended up being very close to the eventual outcome, is recent, has a large number of responses (i.e., thousands or more), and is a live-caller poll. As of this writing, a large, recent poll from Siena/NYT will generally be given a lot of “weight” in this model. The Siena/NYT poll is largely considered the best pollster because they are very transparent about their methodology, conduct polls with massive numbers of people surveyed, and generally work hard to explain their outcomes and own their faults. All of which we love.
We also want to include data about other, miscellaneous things that we know tend to sway elections. You know that you can make alterations like leave out cilantro or add a ¼ tsp more salt to taste, but if you remove the mayo from fry sauce, you’re just eating ketchup with pickles.
For a poll, these fundamental ingredients are things like how the economy is doing. A good economy tends to benefit incumbents. What people think of the president. If the president is, say, a Democrat that everyone hates, Democrats will tend to do poorly in the next House and the Senate races. Whether or not it’s a midterm election. The incumbent President’s party tends to do worse overall in midterms. And historical, correlated polling errors. If the polls systematically under- or over-estimate non-college white voters, then Pennsylvania, Michigan, and Wisconsin are all likely to have significant polling errors.
Understanding what goes into the pot keeps you from swallowing bad statistics whole. When you know that an election model is just a carefully weighted recipe, you stop panicking and start understanding. All of this data is useful as we attempt to predict what the polls will do! This matters doubly for us here in Utah, which is a polling desert. When a rare local poll does drop showing our governor and elected leaders losing support, the group chat instantly explodes. In a state where polls are as rare as rain, one single poll is just a splash of vinegar that adds brightness, not the whole batch of fry sauce.
How It Works: The Stupid Version
Let’s test our theory, then.
One simple thing we could do to test how the electorate is leaning is to simply average the margins of all the polls we have.
A Simple Polling Average
Example polls
YouGov
Campaign internal
Siena/NYT
Local Research Co.
Open online poll
Average of 1 poll: R+1.0
Imagine you have five polls in front of you. If you just add them up and take a quick average, you get Democrats ahead by 1.8 points. In other words, this “model” “predicts” that the electorate will generically lean Democratic by 1.8 points.
How It Works: The Less Stupid Version
Of course, this almost certainly is not a very good model. In the first place, not all polls are equally good. We want some polls to count for more aka have more “weight”. We also want to punish bad polls like internal campaign polls that skew towards a candidate and other pollsters who have bad methodology and are using polling as a means to influence voter or donor behavior.
Because of historical data, we know the following about polls:
- Bigger polls are (mostly) better. The more people you ask, the less chance there is that any individual mistake you make seriously impacts the outcome.
- Recent polls are (mostly) better. The electorate can (and does!) change their opinion frequently. Especially after a big event, like a debate (see: Joe Biden).
- Some pollsters are better than others. NYT/Siena is excellent, and its long track record should count for more than newcomers.
- Partisan polls and internal polls are (mostly) worse. If a poll is conducted by a campaign, that should count for less than a serious, non-partisan pollster.
- Many polls in quick succession should not tilt the average too much. Outfits like YouGov sometimes run many, many polls at once, and if we counted them all at full weight, they’d overwhelm the average.
Now, let’s go back to our example. Suppose that each poll now also has a “weight”, like a grade, which is a combination of the above factors. An excellent poll might get a 100% or A+ grade, while an abysmal poll might get a 10% or F grade. In other words, the F poll here counts for 10% of what the A+ grade counts for.
So let’s give each poll a weight (or “grade”) and compute the weighted average:
- D+5 gets 10% (an F)
- R+1 gets 90% (an A-)
- D+3 gets 80% (a B-)
- R+2 gets 95% (an A)
- D+4 gets 10% (an F)
A Weighted Polling Average
Weighted polls
Campaign internal
YouGov
Siena/NYT
Local Research Co.
Open online poll
Weighted average of 1 poll: D+5.0
The weighted average is (5*0.1-1*0.9+3*0.8-2*0.95+4*0.1) / (0.1+0.9+0.8+0.95+0.1) = 0.5 / 2.85, or D+0.2 rounded to one decimal place.
What this tells us is if we consider all polls equal, this polling model says Democrats are ahead, at D+1.8.
If we consider some polls to be better than others, this polling model shows a near tie, with Democrats ahead at D+0.2.
This is a huge difference. It’s the difference between a relatively close victory and a nailbiter. And we get it by incorporating a little bit of historical data into the polling model! This is the fry sauce of modeling - a tiny shift in how you measure what you add to your recipe can completely alter whether a race will be a secure victory for Democrats with no need for external investment, or a 3-alarm house fire with all hands on deck nationally.
It Is Impossible to Predict Elections.
Now that we have a nice sense of how the models work, let’s look at an actual model, back-tested against the 538 archives.
Here is the formula we use to calculate the polling average. It roughly does what we talked about in the last section, assigning each poll a weight (“grade”).
average margin =
sum(poll weight * poll margin) / sum(poll weight)
poll weight =
recency
* sample size weight
* pollster quality weight
* population weight
* anti-flooding weight
* partisan/internal weight
Let’s put those choices to work. Below, try to find one set of Tiny Model weights that gets every election within one percentage point of the actual national House vote. The same weights apply to every year. Watch the largest remaining miss: bringing the average error down does not fix every election.
The highlighted outlier years are the ones where 538’s average missed the final House vote by at least four percentage points one week before Election Day. That group stays fixed as you change the weights. Every year stays visible, with blue dots showing 538 at the selected cutoff alongside your gold dots, so you can see which ones are still outside the target.
Closer Than 538 counts the years where your weights do better, and each row shows how many points closer or farther you are. That is a separate question from getting every year right: you can beat 538 and still miss the result.
Try the sliders, then click Show the Best Attempt to apply the settings with the smallest worst-year error, precomputed across all the settings available here. Open Detailed Breakdown for the outlier and other-year scorecards, individual polling averages, and 538 comparisons. Missing years stay unavailable; they never count as fixed.
Play around with it! Just remember: tuning the recipe to elections we already know is not the same as predicting the next one.
Try to Fix the Model! (And beat 538!)
Can you bring every year within 1 percentage point of the actual result? Move the sliders and watch which years still miss.
Adjust Tiny Model
A poll loses half its recency weight after this many days.
How much larger samples count. Standard uses the square root of sample size.
How much repeat polls from the same pollster are down-weighted within 14 days.
Off by default. Turning this on uses ratings that include later elections.
Pollster ratings are off: later performance does not influence these weights.
Each slider stop uses a result calculated by the Python model, not an interpolated estimate. Half, Standard, and Strong apply exponents of 0.5, 1, and 2 to the usual weighting factor. Off gives that factor a weight of 1.
| Year | Absolute error, zero to 15 points | Off By, Your Weightsvs. 538 |
|---|---|---|
| 1998 | 2.27 percentage points, outside target, below the 4-point outlier threshold 538: 2.26 percentage points, outside target.<0.01 pp farther compared with 538 | |
| 2000 | 2.71 percentage points, outside target, below the 4-point outlier threshold 538: 2.48 percentage points, outside target.0.23 pp farther compared with 538 | |
| 2002, outlier | 6.67 percentage points, outside target, at or above the 4-point outlier threshold 538: 6.69 percentage points, outside target.0.01 pp closer compared with 538 | |
| 2004, outlier | 5.93 percentage points, outside target, at or above the 4-point outlier threshold 538: 5.78 percentage points, outside target.0.15 pp farther compared with 538 | |
| 2006, outlier | 6.35 percentage points, outside target, at or above the 4-point outlier threshold 538: 8.13 percentage points, outside target.1.78 pp closer compared with 538 | |
| 2008 | 1.32 percentage points, outside target, below the 4-point outlier threshold 538: 0.81 percentage points, within target.0.51 pp farther compared with 538 | |
| 2010 | 0.89 percentage points, within target, below the 4-point outlier threshold 538: 2.64 percentage points, outside target.1.75 pp closer compared with 538 | |
| 2012 | 0.63 percentage points, within target, below the 4-point outlier threshold 538: 0.24 percentage points, within target.0.39 pp farther compared with 538 | |
| 2014, outlier | 5.05 percentage points, outside target, at or above the 4-point outlier threshold 538: 4.65 percentage points, outside target.0.40 pp farther compared with 538 | |
| 2016, outlier | 5.69 percentage points, outside target, at or above the 4-point outlier threshold 538: 4.59 percentage points, outside target.1.10 pp farther compared with 538 | |
| 2018 | 0.89 percentage points, within target, below the 4-point outlier threshold 538: 0.04 percentage points, within target.0.85 pp farther compared with 538 | |
| 2020, outlier | 4.71 percentage points, outside target, at or above the 4-point outlier threshold 538: 4.77 percentage points, outside target.0.06 pp closer compared with 538 | |
| 2022 | 3.01 percentage points, outside target, below the 4-point outlier threshold 538: 1.66 percentage points, outside target.1.35 pp farther compared with 538 |
Precomputed across all 320 available settings, including hindsight ratings, to make the largest miss as small as possible at this cutoff.
Even the best available setting leaves a 6.34-point miss across the 13 years with data.
Detailed Breakdown
Outlier Years
6 of 6 elections
- Your Weights
- 5.73
- Starting Weights
- 5.73
- 538
- 5.77
No change from the starting weights at this precision.
Other Years
7 of 7 elections
- Your Weights
- 1.67
- Starting Weights
- 1.67
- 538
- 1.45
No change from the starting weights at this precision.
Scores use mean absolute error (pp). Your weights, the starting weights, and 538 are scored on the same available elections at the selected cutoff. Starting weights are the settings used when you press Reset.
Same scale for every election and setting. Colors identify the series, not the parties.
1998
Cutoff:
- Tiny Model
- D+1.5
- Error: 2.3 pp
- 538
- D+1.5
- Error: 2.3 pp
- Actual
- R+0.8
43 polls; 36.7 effective. Latest poll: .
538 archive date: .
2000
Cutoff:
- Tiny Model
- D+2.3
- Error: 2.7 pp
- 538
- D+2.1
- Error: 2.5 pp
- Actual
- R+0.4
48 polls; 38.7 effective. Latest poll: .
538 archive date: .
2002
OutlierCutoff:
- Tiny Model
- D+1.6
- Error: 6.7 pp
- 538
- D+1.6
- Error: 6.7 pp
- Actual
- R+5.1
38 polls; 33.4 effective. Latest poll: .
538 archive date: .
2004
OutlierCutoff:
- Tiny Model
- D+3.2
- Error: 5.9 pp
- 538
- D+3.0
- Error: 5.8 pp
- Actual
- R+2.7
56 polls; 46.4 effective. Latest poll: .
538 archive date: .
2006
OutlierCutoff:
- Tiny Model
- D+14.5
- Error: 6.3 pp
- 538
- D+16.3
- Error: 8.1 pp
- Actual
- D+8.2
58 polls; 53.5 effective. Latest poll: .
538 archive date: .
2008
Cutoff:
- Tiny Model
- D+9.7
- Error: 1.3 pp
- 538
- D+10.3
- Error: 0.8 pp
- Actual
- D+11.1
52 polls; 43.1 effective. Latest poll: .
538 archive date: .
2010
Cutoff:
- Tiny Model
- R+5.9
- Error: 0.9 pp
- 538
- R+4.1
- Error: 2.6 pp
- Actual
- R+6.8
53 polls; 44.0 effective. Latest poll: .
538 archive date: .
2012
Cutoff:
- Tiny Model
- D+0.4
- Error: 0.6 pp
- 538
- D+0.8
- Error: 0.2 pp
- Actual
- D+1.0
38 polls; 32.2 effective. Latest poll: .
538 archive date: .
2014
OutlierCutoff:
- Tiny Model
- R+1.1
- Error: 5.0 pp
- 538
- R+1.5
- Error: 4.7 pp
- Actual
- R+6.2
40 polls; 35.9 effective. Latest poll: .
538 archive date: .
2016
OutlierCutoff:
- Tiny Model
- D+4.6
- Error: 5.7 pp
- 538
- D+3.5
- Error: 4.6 pp
- Actual
- R+1.1
59 polls; 50.4 effective. Latest poll: .
538 archive date: .
2018
Cutoff:
- Tiny Model
- D+8.2
- Error: 0.9 pp
- 538
- D+9.2
- Error: 0.0 pp
- Actual
- D+9.1
84 polls; 72.8 effective. Latest poll: .
538 archive date: .
2020
OutlierCutoff:
- Tiny Model
- D+7.8
- Error: 4.7 pp
- 538
- D+7.9
- Error: 4.8 pp
- Actual
- D+3.1
64 polls; 50.1 effective. Latest poll: .
538 archive date: .
2022
Cutoff:
- Tiny Model
- D+0.3
- Error: 3.0 pp
- 538
- R+1.1
- Error: 1.7 pp
- Actual
- R+2.7
139 polls; 106.9 effective. Latest poll: .
538 archive date: .
Data and Comparison Notes
The challenge is to bring every year within 1 percentage point, not to minimize average error. The compact chart shows absolute error on a fixed scale; zero means the estimate matched the actual result. A year is inside the target only when its unrounded error is at most 1 point. Missing years never count as successes. Gold dots show your weights; blue dots show 538 at the same cutoff. The 538 dots stay fixed when you change weights, but follow the selected cutoff. The headline scores and “Off By” column still describe your weights. “Closer Than 538” counts years with smaller absolute error, using only years with both estimates. Each row shows the difference in percentage points (pp). Comparisons use unrounded errors, allowing for floating-point roundoff; ties do not count as wins. Differences smaller than 0.01 pp keep their direction and are labeled “<0.01”.
“Best attempt” is precomputed from every one of the 320 settings offered here and picks the smallest worst-year error on the same available years. Ties keep the first setting in the fixed dataset order. It may turn on hindsight ratings. This is an exhaustive search of this widget's settings, not proof about every possible polling model or every possible weight.
“Outlier” is a fixed label for a 538 error of at least 4 points at the 7-day cutoff. It is not a penalty in the model or a label chosen from your current results. Other years have smaller 538 errors at that reference cutoff. Neither group is reselected when you move a slider or change the cutoff.
The detailed breakdown shows the Democratic minus Republican share of the two-party vote. Tiny Model and 538 averages are normalized as 100 × (D - R) / (D + R) to match the final House results. These are national polling averages, not seat forecasts or predictions of presidential winners.
Tiny Model uses the available historical raw-polls archive; 538 used its own inputs and methodology. This is an output comparison, not a controlled comparison using identical poll sets. Some early cutoffs have very few polls or none at all.
Polls are included by their recorded poll date (the median fieldwork date in this archive), not a verified publication date. The 538 point is the latest archived average on or before the cutoff, no more than seven days old. Source dates are shown for every election. This is a retrospective replay, not a complete reconstruction of information available at the time.
An effective poll count describes how concentrated the weights are, not whether the model is statistically plausible. Trying settings on these elections is exploratory: a better fit here does not demonstrate better performance on a new election.
- FiveThirtyEight raw historical generic-ballot polls
- FiveThirtyEight historical generic-ballot toplines (through 2016)
- FiveThirtyEight generic-ballot averages (original URL)
- FiveThirtyEight averages, archived December 3, 2024 (2018 onward)
- MIT Election Lab House votes via TidyTuesday (recorded D/R votes)
- FiveThirtyEight ratings snapshot (hindsight)
So, Yes. Impossible.
But! We still want to try our hand at predicting the future. Take a look above at our Tiny Model to see how its polling averages compare with 538 and past election results.
In the next article we’ll lay out why we decided to create our own model when there are people (Silver Bulletin, NYTimes, etc.) whose entire job is to create models like these. Why us, why now, and why in a Utah-focused publication?
See you next week for more.