The polls are shit

It all went down in mid-September, at a time when the FiveThirtyEight polling averages showed the slightest of leads for Kamala Harris in North Carolina, a must-win state for Trump. Her edge was short-lived: The averages moved back to favoring Trump. And Quantus Insights, a GOP-friendly polling firm, took credit for this development. When a MAGA influencer celebrated the pro-Trump shift on X (formerly Twitter), Quantus’s account responded: “You’re welcome.”

[...]

A flood of GOP-aligned polls has been released for the precise purpose of influencing the polling averages, and thus the election forecasts, in Trump’s favor. In the view of these critics, the Quantus example (the firm subsequently denied any such intent) only made all this more overt: Dozens of such polls have been released since then, and they are in no small part responsible for tipping the averages—and the forecasts—toward Trump.

[...]

[C]ritics see all this as a hyper-emboldened version of what happened in 2022, when GOP polls flooded the polling averages and arguably helped make GOP Senate candidates appear stronger than they were, leading to much-vaunted predictions of a “red wave.” Most prominently, Democratic strategist Simon Rosenberg and data analyst Tom Bonier, who were skeptical of such predictions in 2022 and ultimately proved correct, are now warning that all this is happening again.

[...]

They are also engaged in a data-driven psyop designed to spread a sense of doom among Democrats that the election is slipping away from them.

[...]

But the guardians of our nation’s polling averages at FiveThirtyEight, The New York Times, and elsewhere all adamantly deny that GOP polls are seriously harming their averages and forecasts, and they offer their own data-driven case to back that up. So who’s right?

[...]

On the one hand, pollsters undercounted Donald Trump’s vote in 2016 and 2020. On the other, in 2022, some of the averages, fed by GOP data, inspired certain observers to discern the infamous red wave that never materialized. So the question now is: Will 2024 be more like 2016 and 2020, presidential elections in which there was a hidden Trump vote, or will it be more like 2022, a midterm campaign but the first post-Dobbs election, when at least some observers missed the Democratic vote that turned out in no small part in response to the Supreme Court taking away the right to an abortion?

The 2022 cycle also arguably saw a new phenomenon really come to the fore: the rise of openly right-leaning pollsters that consistently showed better results for Republican candidates. Now these questions have once again arisen: Should these pollsters be included in aggregators’ averages or not? And what should you think of the case for their inclusion made by the aggregators, which is that they weight polls in a way that reflects their comparative credibility?

  The New Republic

I don't think I'm buying it.

It’s worth understanding why aggregators see value in averaging the polls in the first place. The basic premise behind the idea—and behind including as many polls as possible in those averages—is that the more data one has, the more likely the polling is to offer a reasonably accurate picture of a race.

Even if it's fallacious data?

Something like this happened in 2022: As Nate Cohn wrote for the Times on the eve of that election, the averages were being bombarded by “a wave of polls” from firms that didn’t “adhere to industry standards for transparency or data collection” and which were producing “much more Republican-friendly results.” Democrats ended up defying the results suggested by some of the averages, picking up a Senate seat and holding House losses to a minimum—itself a historically anomalous result for a party holding the White House in a midterm election—even as many predicted a GOP rout.

Some of the pollsters that got those races wrong are the same ones pumping out polls right now on the presidential race.

[...]

The keepers of the averages insist that the impact is very minimal. Outfits like FiveThirtyEight; Split Ticket, the Times’ in-house polling tracker; and Nate Silver’s forecast all take methodological steps ostensibly to ensure that “garbage-in” polls don’t lead to “garbage-out” results. These include downgrading the “weight” of polls thought to be systematically biased so they have less influence on the averages than high-quality polls do. (FiveThirtyEight has detailed criteria for determining whether pollsters are high quality, including empirical accuracy and methodological transparency.) Another step is adjusting for a particular pollster’s “house effects” to downplay biases.

[...]

G. Elliott Morris, who runs FiveThirtyEight, recently calculated that if the averages only include high-quality polls—and not GOP-aligned ones—the results are in some states less than one-half a point different. The Times’ Cohn, who recently acknowledged that we’re seeing a “deluge of polls from Republican-leaning firms” in the averages, ran a similar calculation and found the results moving only imperceptibly.

[...]

But all this raises another question: Why include GOP-leaning polls in the averages in the first place?

Those doing the polling averages do offer a nontrivial argument for including them. It’s that having more polls—particularly now, when polling is very expensive and news outlets are doing it less often—allows one to track the trajectory of the race more closely, says Lakshya Jain, co-founder of Split Ticket. The result, he says, is that properly weighted data from those firms adds more value than it subtracts.

What’s more, casting out pollsters raises other methodological challenges: Where exactly do you draw the line between a GOP-leaning pollster whose data is somewhat biased but still valuable if weighted properly and one who is producing data that’s so beyond the pale that it should be excluded?

[...]

“You generally don’t want to throw data out if you can avoid it,” Jain says. “You just have to treat it carefully.” Jain notes that “throwing data out” risks not being “honest” as well, in the sense that this decision could be influenced by a different form of “bias.”

So....we'll take your data, we just won't give it any credibility is better? Sounds a little dishonest, actually.

Rosenberg and Bonier, the leading critics of these polling aggregations, are quick to point out that even shifts of a small magnitude produced by GOP polls risk badly misleading people.

[...]

How much does that matter? In the aforementioned calculations run by FiveThirtyEight’s Morris, he posits that the polling averages in Pennsylvania that include the GOP-aligned firms are around 0.8 points more favorable to Trump than the ones that don’t.

That seems small, but as of October 22, FiveThirtyEight’s overall averages had Trump ahead in the state by 0.3 points. So it’s reasonable to assume that without the 0.8 points awarded to Trump partly by the inclusion of GOP-friendly polls, Harris—and not Trump—might be narrowly ahead.

In the real world of media spin wars, that sort of difference does matter. In the last week or so, when the averages edged toward Trump, both TV commentators and Twitter accounts cited the tiniest of leads for Trump as evidence that he’s currently winning the state. Even more irresponsibly, some outlets assign candidates electoral votes based on such narrow leads. The GOP polls nudged the averages by less than a point, but they also arguably moved them in a way that prompted people to declare that Trump is now winning—not even just leading, but winning—the election.

[...]

Jain also acknowledges that the argument for removing low-quality GOP pollsters from the averages is reasonable. “I may not agree with it, but I think that’s a totally valid and defensible methodological approach,” he said. Still, Jain thinks including adjusted data is a better choice.

Agree to disagree?

All of this can have serious real-world consequences. As the Times reported after the 2022 elections, red-wave-polling-fueled perceptions of the races ended up producing pessimism even among Democratic operatives, leading to a situation where candidates in potentially winnable races were denied party resources, possibly influencing outcomes.

[...]

Cornell Belcher, the Democratic pollster often seen on MSNBC, thinks it’s obvious that the polls are being gamed, and that this matters a lot. “Are you fucking kidding me?” he said. “I said this several years ago. I’m glad it’s catching on.” He sees GOP firms as using polling not simply to deliver accurate assessments of where races stand but “to drive fundraising and outward narratives just as much as internal strategies.”

[...]

Michael Steele, the former Republican Party chairman who is strongly anti-Trump but has retained his GOP registration, [said] pro-Trump pollsters are feeding a perception of a Trump lead to provide his allies a way to blame a Trump loss on a rigged election. “They’re gamed on the back end so MAGA can make the claim that the election was stolen.”

[...]

[T]hat’s how you lead people to think the race was stolen,” [Stuart Stevens, the anti-Trump former Republican pollster,] said. All manner of postelection mischief-making, and maybe even political violence, would be thus justified.

[...]

We now know that the wave of data coming from some right-leaning pollsters is posing serious methodological challenges. We also know that these pollsters are influencing the discourse in a way that risks misleading people. How off the mark all this data really will prove to be remains to be seen. But the question of whether operatives, journalists, and news consumers are going to let that data shake up our perceptions in the present—well, that’s very much within our control. So let’s choose not to do this instead.

Meanwhile...