"I don't believe the polls" is an almost mandatory declaration for candidates who think they're behind. But, as another campaign season heats up, the rest of us, too, might be wise to view polls with increased skepticism.
Why polls can't be trusted anymore
Widespread reliance on cellphones increases costs, uncertainty.
July 17, 2015 at 10:49PM
Opinion polls, for all their painstakingly scientific intentions, find themselves on shaky ground. In the recent Greek, British and Israeli elections, they showed tight races that turned out lopsided. In the U.S. last year, a number of polls missed the Republican wave that delivered sizable GOP majorities in both houses of Congress.
Three trends appear to be driving the problem: peoples' growing reliance on cellphones; their unwillingness to answer questions and their unpredictability (especially in the U.S.) in showing up to vote. All of that spells more and more uncertainty for polling.
The only fix is to draw random samples from larger populations, but that sends the cost of high-quality polling through the roof. Not so many years ago a reliable poll required the dialing of 2,000 or 3,000 land line phone numbers to draw a valid sample of 1,000 people. It produced a scientifically accurate result to within 3 percentage points either way. If Jones led Smith by 48 to 44 percent, you could be reasonably confident that Jones was slightly ahead. Now, with three-fifths of Americans relying on cellphones, separate samples must be assembled for land line and cell populations. With the automated dialing of cellphones prohibited, gathering a valid sample consumes far more time, labor and cost.
To further complicate matters, caller ID and automated answering devices help people avoid unwanted calls. Response rates that were 80 percent in the 1970s have fallen below 10 percent now. Altogether, it's not unusual for pollsters to have to dial 20,000 to 30,000 numbers to complete a 1,000-person survey.
One consequence is that some pollsters take risky shortcuts that are cheap but far less reliable. Another is the proliferation of "pretend polls" that have no scientific basis at all. A conservative website asking its readers to evaluate Obamacare, for example, produces a heavily biased result that no one should take seriously.
The bottom line is that genuine, high-quality polling is harder to find. For the chattering classes, this new reality requires a big adjustment. More than we like to admit, our public discourse — and our journalism — relies on polling evidence that's not as reliable as we assume. Tough questions should be asked of every poll. Is the sample truly random? What method was used? (Automated dialing and Internet polling aren't to be trusted.) Do cellphones make up one-half to two-thirds of the sample? (They should.) How many people were called and how many were interviewed? (The more the better.)
"Polls can no longer make fine distinctions," advised Cliff Zukin, a polling expert at Rutgers University. People can trust polls on broad issues — like what voters are most concerned about — but not so much on the political horse race, he cautioned. People should think less about head-to-head races among candidates and more about broad groupings — those who appear to be at the top, in the middle and at the bottom.
There may come a day when polling discovers statistical techniques that regain the public trust, but that day hasn't yet arrived.
Now that Gov. Tim Walz’s vice presidential bid has ended, there’s important work to do at home. Reinvigorating that “One Minnesota” campaign is a must.
