Why You Probably Shouldn’t Trust This Week’s Political Polls
This past week in Iowa, Ted Cruz took the lead away from Donald Trump, the floofy-haired presidential candidate who’s been dominating polls and political coverage for the past six months. Unsurprisingly, this made news—polls provide numbers that promise to objectively quantify just how well a candidate is doing. But a lot of methodology goes into these polls, and that affects how much voters (and mere political spectators) should trust them. Here’s how pollsters arrive at these numbers, and why they’re often not as accurate as they could be.
Caveats, Caveats, and More Caveats
First, some clarification: The purpose of a poll isn’t to predict what’s going to happen. Instead, a poll is like a barometer of public opinion that tries to measure, as accurately as possible, how people feel about candidates during a short window of time. Obviously, those emotions have a huge bearing on who wins elections in the end, so they’re what most pundits use to predict the outcome of an election.
Now, it’s impossible to predict the future, but some guesses are better than others. Earlier polls are less likely to be predictive, since there’s more time between the poll and the actual election for some game-changing event (scandal, economic collapse, meteorite obliterating D.C.) to occur. Or candidates could drop out of the race, freeing up potential voters to throw their support to someone else. Then there are human foibles that can affect results: People might say they’re going to vote, and then don’t.
But say you’re not interested in all this prognostication. You just want to know how people feel about candidates at that moment—in other words, whether the poll is valid.
The way questions are asked matters, says Ken Dautrich, a survey researcher at the University of Connecticut. People might change their responses based on whether they’re making them in front of a computer versus or talking to actual people (as in a caucus, where party voters gather to discuss and vote on candidates, or to a pollster over the phone). Or they haven’t thought much about which candidate they actually agree with, and simply choose the last name they hear because they feel obligated to respond.
“Polling is about finding and minimizing error,” Dautrich says. Unfortunately, the only source of error statisticians can really quantify is something called margin of error, which refers to how much the results might vary if the pollsters had asked everyone the poll says it represents. This wiggle room is only based on sample size, Dautrich says—it doesn’t take into account any of the other invisible, hard-to-quantify errors mentioned above.
All these things are baked into the messy nature of polling and the political process itself. But the way pollsters are getting data from people is also going through something of a tectonic shift.
If you wanted to run a good poll, the first thing you’d need is a nice, random, representative swath of people to quiz. Until the early 2000s, this was relatively easy to get: Pollsters would pick names and numbers randomly out of a phone book, call people up, and chat. That way, statisticians only had to tweak their data a little to get results that were more or less a snapshot of the nation’s views.
Now, though, the tech this system depends on has fallen out of favor. People stopped picking up their phones when robo-telemarketers and caller ID appeared, and now around 44 percent of US households don’t have landlines at all. The people who do tend to be older and retired, a specific slice of the population that skews results. Cell phones aren’t tied to geographic locations as closely (they are mobile, after all). Not that polling companies would be able to call most people anyway—conducting these surveys is prohibitively expensive.
Pollsters are getting a little apocalyptic about all this. “On average, the quality of polls today is much, much lower than it was a decade ago,” Dautrich says. In response to its inaccurate 2012 showing, polling giant Gallup announced in October that it doesn’t plan to conduct support polls for the 2016 primaries, and maybe even the general election. “The question I’m constantly obsessed with is, will polling fall off a cliff one day, or will it be more gradual?” says David Rothschild, a Microsoft economist who studies polling.
Instead, news and polling organizations have turned to the Internet, where polls are quick to set up and administer and cheap to host. It makes the statistics trickier, though: Pollsters have no way of making sure that the people who visit their poll are representative of American voters as a whole.
Unlike traditional pollsters, Rothschild is optimistic about the future of polling. “It’s a trade-off between representativeness and depth of information,” he says. With more data, analysts can create much more adaptive, detailed polls, in which different people are asked different questions depending on who they are. Instead of a snapshot of people’s views, Internet-based polling could spit out continuous feedback to see how people react to things in real time.
And more data means more analytics (and growing stables of data scientists to run them)—both to make sense of all the numbers and to figure out how the data relates to the general population. Different companies are developing their own specific systems, and it’s hard to judge who’s actually onto something. But soon, Rothschild thinks, a few good methodologies will settle out.
So if you’re a Cruz supporter, don’t go celebrating too soon. (Trump is ahead in at least one poll, anyway.) But polls aren’t useless, either. Done right and taken on their own terms, they’re great for indicating sentiment and building a sense of how others are feeling. Unless you’d rather not know how your least-favorite candidate is doing, in which case you’re probably sunk until next November.