About Biden’s lead in state vs national polls (I’ve been saying this for weeks!) 📊 May 17, 2020
Plus, a few paragraphs on what to make of “battleground” polls
Welcome! I’m G. Elliott Morris, a data journalist and political analyst who mostly covers polls, elections, and political science. Happy Sunday! This is my weekly email where I write about politics using data and share links to what I’ve been reading and writing.
Thoughts? Drop me a line (or just respond to this email). Like what you’re reading? Tap the ❤️ below the title and share with your friends! If you want more content, I publish subscriber-only posts 1-2x a week.
I have some more thoughts on electoral handicapping for you this week. When it comes to presidential election polling, it’s important to know what to look for to separate the signal from the noise. Recent analysts have shared some data on both state-level and “battleground” polls that require a second look in order to extract such signals.
As ever, if you have related thoughts, I’m only an email away. I’m particularly open to new ideas for blog posts right now. Don’t be shy.
About Biden’s lead in state vs national polls (I’ve been saying this for weeks!)
Plus a few paragraphs on what to make of “battleground” polls
CNN election analyst Harry Enten has a new piece online today entitled “State 2020 polls suggest Joe Biden has a clear national lead.” In it, he writes:
In the competitive states (where most of the state polling has been conducted), there has been an average swing of 6 points toward Biden compared to Clinton's 2016 result. The same is true in the non-competitive states. [….] Biden has posted leads of greater than 5 points in places like Michigan and Pennsylvania. He is ahead in more than enough states to capture 270 electoral votes, if the election were held today.
This ought to sound familiar to you dutiful readers who have been following along with my blog posts over the past month. I wrote two weeks ago about this very pattern, insights I gather from a “toy” election model I’ve put together for analyzing the polls. And on Twitter last Sunday, I noted that the divergence between national and state polls was the same margin Enten that writes about now:
I make this point mostly in jest. It does not matter much who is first to the punch in making a point about electoral handicapping so long as someone makes it. But what Harry and I have both neglected to do so far is answer why state polls are worth taking more seriously than national polls.
Aside from two simple points about sample size and population relevance—50 polls spread across 10-20 states will tell you more about how they vote than 50 national polls—there is the empirical accuracy of the two measures. Below, I’ve plotted predictions for the 2000-2016 presidential elections from two sources: on the left, a model trained to turn national polls into predictions of state results based on each state’s political lean; and on the right, a simple average of polls in each state released in the final month of each election.
It’s hard to see, but state-level polls do a significantly better job at predicting state-level election results than do national polls. The root-mean-square error (RMSE) for national polls is 2.5 percentage points for the Democratic share of the two-party vote, whereas the model has a RMSE of 3.3 points—a roughly 30% reduction in error. Given you have a rough idea of how many voters each state will contribution to the nationwide electorate, these polls also give you a better prediction of the national vote breakdown than do national polls. So it’s important that Biden is beating his national margin in the state-level data because the rosier data for him may be closer to the eventual results than national polls.
People hypothesize that Biden’s overperformance in state-level data is due to a variety of factors. Some are more evidenced than others. First, there has been a higher incidence of online polls at the state level this year, and online panels often overstate results for Democrats. That could explain the pattern. But this is not always the case. Importantly, Biden’s higher state versus national lead remains even if you only analyze polls conducted over the phone by live interviewers, which analysts often think are more accurate (though again, it’s not clear this is true this year).
There is also the possibility that state-level polls have been improperly weighted. In 2016, many pollsters at the state level failed to weight for the education of their respondents, overstating the share of college-educated voters in the electorate and thus overestimating support for Hillary Clinton. But many pollsters have fixed these issues since the last election (though some have done a better job than others). Even if I subset the analysis to polls that were conducted by a live interviewers AND weight by all the right variables (sex, race, age, education, and sometimes population density), Biden’s lead in state-level polls is still slightly larger than the one implied by national polls (though the gap falls from two points to one).
So the explanation I’ve settled on is twofold. First, there is random variation in the data that may impact this relationship. There are not that many high-quality state polls right now and the delta between them and the national average could float around until we get more polls. But that is not very satisfying, so I also took a look at the states that have been polled so far and compared Hillary Clinton’s margin in them to her margin nationally in 2016. It turns out that the states that have been polled so far are a bit bluer than the nation as a whole—by about 1.5 percentage points. That doesn’t explain the whole delta, but it does explain most of it.
In my toy model, I’ve adjusted for this tendency for state-polls to be different than the country as a whole and still get better predictions for Biden than the national numbers alone. There’s something going on that these factors don’t explain. One possibility could be that the national pollsters have anti-Biden house effects that state polls don’t, or vice versa.
Nevertheless, with history as our guide, the best guess is that the state polls will be more predictive of state-level outcomes than national polls are. And right now, that’s good news for Biden.
Separately, I sent a tweet yesterday about so-called “battleground” state polls, which I don’t find that useful. I got a lot of questions about this so I’ll explain my opinion here.
Peter Baker @peterbakernytBiden has not fully capitalized on Trump's weakness. He leads the president by 5 percentage points nationally in a new @CNN poll, but trails by 7 points among voters in crucial battleground states. @GioRussonello https://t.co/uPEohxh6if
On background: some pollsters and media outlets conduct and report on polls of select competitive states—so-called “battleground” polls. This sounds like it could be useful. If a poll can tell us that one candidate has a slimmer margin in the states we know will decide the electoral college than they do nationally, we can mentally adjust our inferences of electoral competitiveness based on other national polls. But in practice, they are not this useful.
One problem is that some battleground “polls” are actually just subsets of national data to people who live in competitive states. But this introduces some huge sources of bias if the new “poll” isn’t weighted again to be representative of the target states.
Another problem is that even the correctly-conducted battleground polls don’t fit neatly into our models. Election forecasts typically depend on national and state data, and there’s not typically enough extra data to incorporate other hierarchies (such as a group of competitive states). This would require all battleground polls to be conducted among the same states, which they are not; selection varies by pollster. So you’d almost always prefer to have 2 polls of 400 people in 2 different competitive states than 1 battleground poll of 800 people spread across them.
Posts for subscribers
What I'm Reading and Working On
I’m in a modeling headspace right now, so not much to report on the reading front. I am working on a piece about how election forecasting helps us understand politics and political processes, though, so welcome your suggestions for related works.
Thanks for reading!
If you want more content, I publish subscribers-only posts on Substack 1-3 times each week. Sign up today for $5/month (or $50/year) by clicking on the following button. Even if you don't want the extra posts, the funds go toward supporting the time spent writing this free, weekly letter. Your support makes this all possible!