The polls were wrong – sorta. But is it too soon to 'blow up' the polling industry?

Ryan W. Miller, USA TODAY
·11 min read

Yet again, the polls were all wrong, some pundits declared.

"The polling industry is a wreck, and should be blown up," proclaimed Politico Playbook, a newsletter emblematic of "inside the Beltway" conversations, on Wednesday morning.

After a massive miss in 2016, when Democratic nominee Hillary Clinton was pegged as the favorite over long shot Donald Trump, pollsters did it again this year, many said. Democrat Joe Biden was supposed to win by margins of 8 percentage points. Polls in Wisconsin, where the 2016 errors were among the most glaring, had Biden up big.

Despite the conversation and rhetoric around the polling industry, the picture of how far off preelection polls were in comparison with the actual vote is a bit murkier.

Live election results: See the electoral map here

In Wisconsin, Ohio and Florida, there was a clear underestimation of Trump support. In other battleground states, preelection polls were right on the money. The true scope and cause of error won't be known until all votes are counted and a post-election analysis is done.

The American Association for Public Opinion Research, an organization of survey professionals, cautioned against "hasty conclusions based on incomplete returns."

"The results of preelection polls do vary from the current vote results in some battleground states. When all the votes are counted, some of the hundreds of polls conducted on this election will come close to the final vote percentages, but some will not. This has been the case in every election. The issue is how the polls collectively performed in describing the official results of the 2020 election," the group said in a statement.

Regardless, a reckoning with the disconnect many Americans feel between the pollsters and the actual vote may be needed, some pollsters said.

There are "states where it looks like we have these really bad performances," said Joshua Dyck, director for the Center for Public Opinion at UMass Lowell. "On the one hand, we have this big polling error (in Wisconsin, Ohio and Florida), and on the other hand, how many times has (pollster) Ann Selzer been right in Iowa?"

Election results updates: Biden is poised to win but it isn't over. What we know

Where were the polls wrong and did they underestimate Trump's support?

"It's pretty clear that the support for Donald Trump was underestimated in national and in many state polls," said Michael Traugott, a research professor at the University of Michigan's Center for Political Studies.

The full extent of that underestimation will take time to understand, Traugott and Dyck said. Polling data that showed Biden with an average lead of 7.2 percentage points, according to RealClearPolitics' average, and 8.4 points, according to FiveThirtyEight's average, could be wrong.

How wrong matters in terms of historical context, Dyck said.

If Biden wins by 5% in the popular vote, a 3% miss could be within the expected margin of error. In 2012, Obama's win was underestimated by about 3 points.

"It's not great but the thing that's at least heartening about that is that that error can go both ways," Dyck said.

In 2016, national preelection polls weren't that far off from the popular vote, either. Clinton was up by about 3 points before Election Day, per RealClearPolitics, and won the popular vote by 2%.

State level data is where 2016 missed on Trump's support, which was enough to win states that tilted the Electoral College in his favor. A post-election analysis by the American Association for Public Opinion Research concluded that state-level polling before Election Day "clearly underestimated Trump’s support in the Upper Midwest."

That makes the error in Wisconsin all the more glaring. FiveThirtyEight showed Biden's lead in the Badger State at almost 8.5 points. RealClearPolitics showed his polling lead as more than 6.5 points. Days before the election, a poll released by The New York Times and Siena College of more than 1,200 likely voters in Wisconsin showed Biden leading the race by a whopping 11 points.

Votes are still being counted, many of which are mail-in ballots that may boost Biden's lead after news organizations forecast him to win Wisconsin's 10 electoral votes. After about 99% of votes were counted, Biden led by a fraction of a percent Thursday afternoon.

Other states with discrepancies between preelection polling numbers and votes include:

"There's a regional problem here," said Sean Trende, a senior elections analyst for RealClearPolitics. "Something is in the water. I'm not sure exactly what. The Midwest is proving hard to poll."

Some state polls before Election Day were much more in line with the vote totals:

  • Georgia: Margins were thin before Election Day. RealClearPolitics' average had Trump up by 1 point. FiveThirtyEight was flipped with Biden up 1.2 points. The race hadn't been called as of Thursday afternoon, but Trump led by less than half a percentage point after 99% of the vote was counted.

  • Arizona: RealClearPolitics' average was Biden up by less than 1 point. FiveThirtyEight showed Biden up about 2.5 points. Some news organizations called the race early for Biden, but it was still tight Thursday afternoon when Biden led by 2.3 points after 88% of votes were counted.

In Iowa, a local pollster exemplified how some polls were right in 2020, Dyck said. Selzer's data, part of the Des Moines Register/Mediacom Iowa Poll, showed a Trump lead of 7 points in the final days of October. (The Des Moines Register is part of the USA TODAY Network.)

Compared with a RealClearPolitics' average of a 2-point Trump lead and FiveThirtyEight's average of a 1.3-point lead, the Iowa poll drew criticism before the votes rolled in. The poll conducted by Selzer has been a Register exclusive since 1943 and lauded nationally for its accuracy.

“Everybody starts saying, ‘She’s terrible, she’ll be fired, pay no attention,’” Selzer told the Register. "This has happened to me many, many times."

Trump won Iowa by 8.2 points after 99% of the vote was counted Thursday.

Selzer's data captured the Senate race between Republican Sen. Joni Ernst and Democratic challenger Theresa Greenfield. RealClearPolitics' average of the polls had Ernst up by 1.4 points. In the Iowa poll, Ernst was favored by 46% to Greenfield's 42%. Preliminary results Wednesday showed Ernst winning by a 6.7-point difference.

Election results: Why networks are reporting different electoral college totals

Are polling errors in 2020 a repeat of 2016?

Preelection polls capture a moment in time. They're a snapshot of where the "likely electorate" stands. To get to the numbers that are often repeated in the news, it's not as simple as asking voters whom they like, then dividing by the total number of voters polled and getting a percentage.

Pollsters – who reach their pool of people via home phone, cellphone and online survey – make assumptions because the pool is not always representative of the demographics of the electorate.

"The whole concept of the 'likely electorate' is an estimation problem because it doesn't exist until the whole thing is over," Traugott said.

How pollsters go from their larger data pool to a "likely voter" is an informed guess, said Courtney Kennedy, the Pew Research Center's director of survey research. That process is "ultimately part art and part science," she said.

Pollsters need to be more transparent about their "likely voter" models, Dyck said.

"That process is incredibly opaque," he said. "For an industry that prides itself on transparency, there is often no information by, I would say, the majority of pollsters of how likely voters are selected."

The errors in preelection polling data from 2016 have been attributed to many causes, but one that pollsters and political analysts almost all repeat is how education was weighted in poll data.

Weighting occurs when a pollster makes an adjustment to data to try to make it more representative of the population, according to the American Association for Public Opinion Research. "Weighting is used to adjust the relative contribution of the respondents, but it does not involve any changes to the actual answers to survey questions."

In 2016, state polls in the Upper Midwest probably did not adequately account for the overrepresentation of college graduates in their surveys that correlated with an overestimation of support for Clinton, the American Association for Public Opinion Research's postmortem report concluded.

Pollsters vowed 2020 would be different. Adjustments were made, more data came in, closer to Election Day and from higher-quality, trusted sources, pollsters said.

Traugott said a full examination of whether the 2016 errors were fixed and what new problems could have arisen in 2020 will take time and rigorous peer review.

He and Dyck hypothesized that polls may have been affected by the change in how people voted during the COVID-19 pandemic.

Every year, pollsters have to assume how much of the vote will come from early voters and how much will come on Election Day. In 2020, the share of mail-in voters and voters who cast their ballot at early polling sites soared: About 100 million people voted early.

"Outside of a few historically vote-by-mail states, pollsters do not have any experience modeling the vote in this type of election," Kennedy, from Pew, said. "The number of unknowns this year are greater than in any election in recent memory."

Preelection polling data helped Americans understand how those early votes (in general but not in all states) would disproportionately come from Biden voters while in-person Election Day voters leaned toward Trump, Dyck said.

Dyck suggested that pollsters' data was too heavily skewed in favor of the early and mail-in Biden voters because they were more likely to be "at home" and taking heightened pandemic precautions when a pollster's call came in or they saw a survey.

Trende, of RealClearPolitics, was skeptical and suggested there's a deeper problem in polling and trust among Trump voters, as similar errors have occurred in each election since 2016.

"One of the defining characteristics of Trump support has been low levels of social trust. People who think American institutions don't work for them. Those are the people that don't respond to a poll," Trende said. "The problem is you can't weight on an attitude" without polling on social trust.

Is polling dead?

Preelection polling is unique from other public opinion surveys because there is an external measure – the actual vote – that can validate the data, Traugott pointed out.

Polling data is used year-round, election season or otherwise, and captures public sentiments and attitudes in ways that don't show just the "horse race" of which candidate is ahead.

"Blowing up the polling industry is just crazy," Traugott said.

He pointed to possible alternatives: Estimating public sentiment from social media. Analyzing letters to the editor or emails to congressional staffs from constituents. Those measures simply wouldn't compete with the way modern polls are conducted, he said.

Explaining the potential errors of 2020 after 2016 polls faced backlash for not correctly estimating Trump support will be no easy task.

"To suggest to the public that it looks like the same problem (as 2016) but the causes are different would be hard for the average citizen to understand," Traugott said.

Trende was skeptical of how much education around polling methodologies will rebuild trust. "This is not Americans are stupid," he said. "It's that Americans have lots of other things to worry about."

Nate Silver of FiveThirtyEight wrote an article the Monday before Election Day that was headlined, "I’m Here To Remind You That Trump Can Still Win," explaining how the probabilities of the site's models could play out. The New York Times' statistics blog, The Upshot, retooled its election night "needle" to include three needles for key states to better show how the presidential race was going.

Dyck acknowledged errors but noted that polls conducted by trusted, often regional pollsters got it right.

"The result that we're looking at is not like this crazy result out of left field," he said.

Contributing: Katie Akin, Des Moines Register

Follow USA TODAY's Ryan Miller on Twitter @RyanW_Miller

This article originally appeared on USA TODAY: Election polls underestimated Trump over Biden. Does polling work?