Go back

From Polling to Prediction Markets: How Reliable Can We Predict Elections?

from Harvard Data Science Review Podcast

35m 58s

From Polling to Prediction Markets: How Reliable Can We Predict Elections?

This podcast explores the evolving landscape of election polling, highlighting both its strengths and persistent challenges. Hosted by Harvard Data Science Review’s Shally Mangini, it features leading experts Anthony Salvanto of CBS News and Michael Bailey, author of *Polling at a Crossroads*, who discuss how modern polling operates as dynamic modeling rather than static surveys. Key insights include the critical role of turnout data, the difficulty in accurately reaching and representing marginalized voter groups like young people and Hispanics, and the limitations of traditional margin of error calculations. Polling accuracy is now understood as dependent on complex models that account for historical behavior, non-response bias, and shifting voter intentions. The conversation also examines how technology—such as online panels and prediction markets—enhances real-time data collection, with markets offering complementary insights into public confidence and rapid shifts. Ultimately, the discussion emphasizes that polling is not about absolute precision but about capturing the dynamic, human nature of political opinion. The episode concludes with a focus on democratic equity, suggesting that every voter should have a fair opportunity to be heard, even as campaign spending and messaging influence outcomes. The insights are framed as essential for understanding elections and maintaining public trust in democratic processes.

Transcription

6253 Words, 34178 Characters

English
Hello and welcome to the Harvard Data Science Review Podcast. I'm Liberty Vitter, the feature editor of the Harvard Data Science Review. In this month's episode, HDSR Editor-in-Chief, Shally Mangini, are taking a closer look at election polling, how polls are conducted, what they can tell us about voters, and the challenges facing modern survey research. All in preparation for the Harvard Political Analytics Conference 2026, taking place October 8th and 9th in Washington DC. We are joined by two leading voices in the field, who will also be speaking at the conference. Anthony Salvanto, PhD, is CBS News's Director of Elections and Surveys, and really the voice of elections, and Michael A. Bailey, author of polling at a crossroads, rethinking modern survey research and the Colonel William J. Walsh, Professor of the American Government, and the Department of Government and McCourt School of Public Policy at Georgetown University. When we talk about what goes on behind the numbers, the changing landscape of election polling, and what researchers and really the public should keep in mind when interpreting survey results. So let's get into the numbers and what they really mean. So Anthony, walk us through what election night, and let's go with a big election night. So let's go with like, you know, the 2026 midterms, 2028 presidential, whatever it is, a big election night. What does it look like at the CBS News decision desk? What do you know at 7 p.m. that the public doesn't? And what's really the hardest part of calling a race responsibly? Sure. One of the things to start off with is that I'm really proud of the fact that in recent cycles, we've put the decision desk front and center on our broadcast, which is a way of being transparent and going after and addressing some of the questions like the one you raised, how is it that we make projections and what is it that we're seeing? The idea being that we want folks to see things at the same time and just as we see them. And that's also part and parcel of the reason that I present some of that data directly to the viewers. So I'm always grateful for the chance to do that. In terms of specifics, early in the evening, we always start to see the exit poll. And while that doesn't necessarily tell you who's going to win any particular race, it does give you an early sense of the dynamics that are taking shape. That was on voters minds and what types of voters might be inclined to go one way or the other. That starts to set the stage. One of the things I like to talk about or show people is that at the decision desk, we look for patterns when we're describing a race and when we're getting or approaching a projection. What I mean by looking for patterns is even though many of us who stand at the touch screens will point the audience to a particular county, a key county. Watch this one. Watch that one. And that's all fun and that can tell an important story. But to actually make a projection, and this is something I'm sure the listeners of this podcast know, what you're looking for is consistency in the data. And so that will be across many counties or across many precincts. We will be looking for shifts from past vote. We have models that form a ratio estimate of the current vote to the past vote. We evaluate whether or not those models are pointing in a consistent direction. And we will start to get usually multiple models of a race in real time. One of the other details in there is we see differences in vote type. Not just differences by region, geographic region, urban, rural areas, how people vote. But also in the fact that the vote by mail has often differed and often favored Democrats. And the election day vote in recent elections have often favored Republicans. So there's this process at the decision desk of trying to learn from the individual counties from reports what types of votes are being counted as well. And we're transparent about that also. So when all of that comes together and we see a consistent pattern, that's when we can start to make a projection and we've been describing things along the way. It obviously gets really intense as you start to go about eight, nine o'clock eastern and many states's polls have closed. But as I often remind people, it often involves a few cups of coffee and more recently things have been gone late and even into the next day, which just keeps it, keeps it exciting as we go. Anthony, what you just said always reminds me that first of how hard the work you all have been doing, it's incredibly hard. And the part of the reason so hard is that whatever you can do obviously is influenced by the quality of the data, you talk about being consistent. But we also have seen how polls are consistently biased, consistently wrong because the difference between people report, people respond, hope you don't. And that's, I mean, Michael's topic in your wonderful book, pulling at the crossroads and you argue that random sampling paradigm is a factly broke down. How did that problem show up in this cycle? Did the poster handle the better or worse? Then the last three elections, now 2016, 2020, 2024, every election we see this problem persistent. Yeah, I mean, basically, as you alluded to, the polling paradigm of calling of thousand people, they all answer and then you ask them who they're going to vote for. Like, it's amazing. It's amazing, powerful, amazingly powerful. As everyone knows, it's unbiased and we can exactly characterize the degrees of confidence about it. We are light, light, light years away from that, you know, maybe one pollster the New York Times tries to do that. But again, as you guys know, they're getting 1% response rate. And they're on the high end of things. And by the way, like the, and this will get in as we talk more about waiting, you know, like there's a lot of groups that really aren't that hard to know what's going to happen, you know, like, you know, white non-college rural residents or, you know, lawyers in big cities or, you know, like those, those groups are pretty easy, but there's a few groups they're really, really hard. Hispanics tend to be hard and they're moving and maybe moving fast and could determine the election in lots of places. Young people, like they're not going to show up to vote, sadly, in my view, but they're not going to show up to vote. But to what extent do they not show up to vote in the midterm election? And so for those groups, you'd be lucky to have a half a percent response rate. And so they are, and that's what my book is part of a project, is like, we have to really think about when we look at any subgroup, and that's basically the way we're going to solve this problem is we're going to divide the electorate into subgroups. And for each subgroup, we're going to put a number on it. And then we're going to patch together those subgroups. And within those subgroups, that's where things get really, really tricky. And within the groups that are the most interesting ones, it's even harder. So they're really flying pretty blind. And so one of the things you're going to hear a lot of, and it's actually true, is that pollsters do pretty well on average, but my claim is they're doing okay when it's easy, right? So they can kind of, like if you know what's going to happen, you know what's going to happen. But in New York City, is mom Donnie going to get young people to turn out to vote for them? Pollsters said, no, they were wrong. He did way up from the polls. So in general, polls are kind of fine, but like you really have to be careful when it really matters, and that's why it's still going to be an adventure and Anthony's going to have a busy night. He can look at those, those polls going into the election, and you've got the Republicans losing an Idaho and like, do you really believe those polls? Maybe, maybe not, right? And that's going to come down to actually what happens on the ground, precinct by precinct. You know, I think one of the biggest problems that we see with midterms is turnout. You have these levels, right? You have primaries. We have low turnout issues. We have midterms. We have low turnout issues, and then all of a sudden you have the, you know, the 28 election where everybody wants to come out and vote for the president. So how much do you all think that really affects the ability for pollsters to have any idea what is going on? Is it more of a turnout problem than a sampling problem? And is this something that you think there's going to be a way to figure out how to fix in the future? And I'll, Anthony, I'll start with you. Yeah, let me put my polling hat on because I do these, these polls, and this comes back in context to some of what you were just discussing. Let me vigorously defend polling for a moment with this as a, as a guide. First thing to do is separate measurement of attitude and measurement of behavior. There are two different things, of course. What attitude is anything we see in the polls like, you know, how do you think the economy is doing or how do you rate a particular candidate? But behavior is what you're talking about with right, which is, which is turnout. What are you going to do? And there's a measurement difference in that. The attitude, no measurements, of course, we can all talk about questions and certainly some of that will come into modeling or sampling. In the turnout case and behavioral measurement, you can sometimes have an expressed behavior. I'm going to do this, but things get in the way in the human experience, right? You might have a busy day on election day and not make it to the polls, even though you meant what you said when you told the pollster that you would definitely vote. The other part of this is that we have never had so much good data about voters. And this is so important to polling. And quite frankly, it's one of the things that makes polling good when it's good, which is you can take a publicly available voter file. Every campaign has them and everybody uses them. And you can see people's turnout history. You don't know how they voted, of course, but you can see who's turned out. And you can build and you can back test very good models for who is at least based on their history likely to turn out. And so you can, if you choose to and some pollsters do choose to sample or even over sample, folks who are going to be more reliable voters, you can assign a turnout score to every voter in your sample. And you can incorporate that voter history. So every poll you see is a model. And you are seeing that pollsters model of an electorate. Is it easy to do? No, it's not. But one of the things that's really incumbent on all of us pollsters to do is explain to people that these things can vary not simply because of sampling or whether it's a, you know, random sample or not, but also because there are people making these decisions. There are people who have lives and might be on the fence or might not get there. And also because let's not forget there are campaigns out there who are working to change this on the fly. And of course pollsters sort of try out the usual thing. Well, it's a snap shot in time. Yes. But there's a campaign out there that's trying to motivate that person who might have told you the pollster that they are maybe going to vote and they're trying to get them to change that point. And we have to allow for that. In fact, we have to in some way celebrate it because we are measuring something that's ongoing. Otherwise, that's, that's why they have campaigns. So these things can change over time. And we need to tell people that we are capturing part of that dynamic and that it's ongoing. And that's part of the story. So I really, really strongly agree with the Anthony was making that point that that pollsters are modellers now, right? But then as we start unpacking that, I think it makes keeps us in business, by the way, because it's very, very difficult. But one of the first things, again, setting the paradigm of the old pollsters who just did a survey, they wouldn't have to know anything about anybody going into it, right? They would just get a random sample and that would just bubble up and they could just look at that. Instead, they have to know all these things and make these models. And the models are really assumptions. And so two things that I think are really important about this. So one is you have to, and this is literally what's happening and it's every, every pollster is doing this in their shop right now, is you have to say, what's Republican turn out going to be? And like I, I don't know, if we had a random sample of a thousand people, we could just ask questions and then we'd just see, you know, of the Republicans who's going to turn out to vote. And it's a little walkiness of people can change parties and so forth. So it's not trivial, but you would get that here instead. They're going to say, they're going to do a sample of a thousand people and they're going to get, let's just say they get 150 Republicans. No pollster ever would say, ah, I think 15% elector is Republican, they're going to be waiting that group up or down. And how are they going to wait that group up or down? It's more complicated than I'm saying, but the basic ideas they're going to say, well, in general, what's the share of the electorate that is Republican in previous midterms? They're going to take their waiting sampling frame and they're going to put that into their model. And then if they are right, right? And if they have the weights right about what percent is going to be Republican, they're in business and they're going to get it right. And that's what happened a lot like when you go from 2020 to 2024, you can look at 2020, how people turned out when there's Donald Trump running against the Democrat and you can look at 2024. And so 2024 did really well actually in part because like the, the behaviors were really, really quite predictable historically. To take a canonical example where that wasn't the case, and I'm not saying this is like always the case, Obrocco Bamo and he was running in Iowa, the people going into that election had a turnout model of who's going to show up for caucuses and it's in part based on who turned out before and they have pretty good data about who's going to turn out before. And they use those models and then they would predict something that was very, very wrong because the underlying assumptions were no longer true. And that moving target, that's where the, the challenges is to keep up with that moving target. The second point is then within each cell of the sampling frame, again, there's other methods and they don't literally always have cells and sampling frames like I'm talking about, but they're effectively very similar. Within each cell, you have to decide the people who did respond, are they different than the people who didn't respond. So again, like the real interesting groups tend to be Hispanic voters, tend to be non-college white voters, you know, there's not a young people. Those groups are really like really kind of hard to predict or we really, really want to know what's going on with those folks. I really, as a statistician, I really love both of your emphasis about poster is a modeler because, you know, there are some time there are general perception there. Well, you know, this is all signs, but it is trying to be scientific, but there's also lots of arts to it, there's lots of judgment. And I think that's really incredibly important to emphasize. At the same time that I want to ask you a little bit if it's not a confidential about the kind of pooling methods CBS, you know, uses. And the reason I'm asking this is really trying to get your input on the kind of trade-off between these different methods, right? There are these online up in these are probability panels, the trade-off, traditional phone pooling. Now people even talk about using AI to summarizing public opinions. You know, so that's all the tools available. And I want to, you have such experience. I want you to talk a little bit about what did you have learned? What are things come out has a clear trade-off, which is better? Oh, things we should be just, you know, watch out carefully because they all have serious pros and cons. Yeah, I thank you and I'm happy to talk about it as part of the transparency, like I talked about before. The first thing I say is that there are really thoughtful pollsters out there doing good work across various time types of methods. There are folks who are making phone calls to people, often using voter lists like I described, but they're, there are many who are doing good work that way. It's, it's no secret that takes longer and is often more expensive to do, but it's out there. And there are folks out there using text to web methods now. And by and large, and this is something we've done a lot of in more recent years, there are people who use panels. And what I mean by panels is there's a great deal of pre-recruitment in some form that goes on all throughout the year by companies and firms that establish, build, maintain these panels. And these are people in principle who have agreed to be part of the panel and take surveys on it. And what we do and what we make use of from these panels is you can pull a sample from them with known characteristics of those folks. And you can also and we also do it such that you can validate someone can confirm that they're a registered voter, confirm that they live where they live. And then they're part of these panels. And it's very convenient for them to take a survey because they get a secure link or they use an app as part of the panel. The other advantage of that is if we do want to do things like re-contact or see movement, individual change over time, you can if you wish and we do this on occasion go back to that same panelist and look at how they've changed or moved. And what we find is that we not only get very reliable and good data, but we also have the ability to do things very quickly. So we can turn around a representative sample in short order. Now over time, over the years that I've been doing this, some of that I've seen change. And there were times years ago, it's quite a few years now, but we did more pulling on the phone. But the big picture is the industry like all good science evolves and adapts. And if you think in terms of pulling and you think of the frame, the platform that you use to reach people, what you want to do is always be ready and develop capabilities to reach people where they are, to reach people in ways that are convenient for them to express their opinions. And that is part of the of the science of it. And it's about taking advantage of of new technologies in that way. You know, we can't not talk about betting in prediction markets. You know, you have calcium of polymarket. These are legal, heavily traded, and increasingly cited by journalists right alongside polls. Are they a competitor to pulling, a compliment, or are they going to be something else entirely? and what information if any do the markets capture that polls miss or vice versa? I would put them in like kind of a compliment and that says two sides to that. So one side to that is they're going to take basically similar information and then kind of push it. So it can like kind of take the starting point and move one way or the other. But it's also a compliment in the kind of negative sense is that the weaknesses of polls they roll right up into those prediction markets, right? And so it's not. It's kind of fun. It's kind of dangerous in other contexts. But I think in polling, it's kind of harmless in a small net value ad ad say, but I don't think case after case when the polls miss. I don't know that we've had a case where like the betting markets have really, you know, seen beside the scenes and seen what the polls are going to get wrong. I also think compliment the way I look at it is for those of us who study or try to gauge opinion out there, they are an expression of viewpoint for those people who are trading on the markets. We can do well to see what those folks are thinking and what they're willing to trade on. I also think you can use them or look at them to understand what people think of what's call it sort of downstream effects of one race on, you know, on others. So some markets might have will a party take control of the House or the Senate. And then a poll comes out or some rating comes out for one of those races that goes against that grain. Does the market for the overall Senator House control then move, you know, attentively? And that's a way of, again, looking at at least among traders, that kind of confidence that they're expressing in the available information out there. One thing that the prediction markets can definitely do is they can be instantaneous. So if there's some gap or something, you know, or some new news leak of something, then obviously the prediction markets can respond and that's going to take pollsters, you know, a week or so to nail that down. Pooling usually is not just about who's winning, who's losing because it's lots of things about what the vote is thinking, lots of other things, right? Now, initially when I start thinking about these marketing stuff, seems like you're going to shift attention or it becomes winning and losing, that seems like will be a negative, you know, distraction, you know, sense you're losing lots of information are about the voters, behaviors, so on, so forth. But you just reminds me, it could also be again, in terms of, at least for a group of people, seeing how they express their confidence and a micro your point of something being instant, that's actually really important, right? There's a time temporal component is obviously important for all of the things. So it's just like a little bit of follow-up question on this, like do you guys see this, particularly from a research angle? What are things like, for example, for someone like me, as a statistician or for Michael, that we do research, you know, I guess I'm good to see this topic here for PhD students to work on those things. Hey, you know, this is something really quite interesting. So one is there's just a general kind of discussion of what really happens and kind of what this conversation is. I do also want to point out, and I think this just is one of, for me, at least one of the themes as I think about how polling works is that there's margin of errors. And so you're going to see every polling firm is going to have a margin of error. And that margin of error is calculated on a scientific basis for a random sample with 100% response rate. Like, what, like, so I call that poll washing. So that's just a number that you made up in, like, another world, and it kind of maybe sometimes, you know, has some insight or something. It's not like 100% irrelevant. But boy, the scientific basis of the concept of margin of error and the practice of what we're doing is a very big gap. In fact, I mean, I end up being like, I kind of, maybe come off as being like, negative out of polling. I'd say just there's all these vulnerabilities. And in spite of that polling does pretty well. But at the same time, it just is like insane that they're calculated the margin of error based on these assumptions that are just, they're just not there. And or, you know, they use sample size and the bigger the sample size, the better it is, well, some sample sizes are terrible. In a random sampling world, the bigger sampling size is more precise. It is. It 100% is. It's a scientifically undeniable fact and a true random sample. In what we have, and this gets to like the quality issue, you know, like some people have a lot of data and it's terrible. They go, just go get it on the internet. If people click to, you know, rage bait, right? Or they could be from, you know, foreign country or could be bots, right? A big end means nothing, right? It's all in the, has the data been carefully collected and just all the quality controls you have to go into that. So like, as we characterize uncertainty, I think the default that people do, and they kind of feel like they're being sophisticated about looking at margin of error and so forth. I'm just a, honestly, I'm a little wary of that. And I think it's, it's a much harder problem. So I think successfully communicating uncertainty starts with appreciating or understanding how a general audience might come to probability. And we know from studies, and you all of course know that sometimes people take probabilities towards the end of the scale and basically push it to zero or a hundred. And sometimes your heart's in it in a sense of, if one is told there's a 10% chance of something wonderful happening, you might pursue that. Oh, well, that's, you know, we get it, but 10% chance of something horrible happening. And suddenly that 90% goes away and you're thinking, oh, wait a second, I better avoid this situation. That's what I mean by how people interpret that. And what I mean by appreciate it is, how do we find a way into the explanation given those circumstances? The other part of it is, and it's a little counterintuitive for most folks. You all are talking about margin of error and, you know, traditional computations, all that. You're really coming at that by saying, how do we not be colossally wrong? You're not coming at that by saying, everything is going to be exactly precise all the time, which you're doing is narrowing a band of not being wrong. And that's not necessarily how most people approach things or think. And sometimes it's not satisfying for that. People see numbers and they see, they, when we say science around it and they think, oh, it's going to be absolutely precise. But think about how many things in life are just really good estimates. And that's one way to, to, to approach it, to say, look, where it's amazing we can get this kind of an estimate down around this range. And be happy about that. And think that's really cool. I think it is really cool that we can say, you know, that how people feel about given issues or given topics apart from elections. And I think that when people do turn to polls apart from this elections and, and things that you mentioned, I hope. And I think they're looking for the fundamental answer to the fundamental question of who are we? And, and what is this country and the people around me us thinking and doing, which are much larger questions that can be answered by measuring public opinion. And in fact, have to be for, you know, for a functioning democracy. And so that's part of, that's part of the larger. And can I just add one more thing to that? I think as we're thinking about like the air, like just think of Wisconsin, good ministerial, or think of the famous Iowa poll and shelter, Iowa poll with Harris up by eight or whatever the exact number was in the 2024 cycle. Like those, those are just wrong. And I think they're far outside of the, you know, margins, you know, they're six sigma away or whatever. They're, they're vanishingly small probability of happening under conventional depictions of margin of air. And so that then helps us, I think, put it in the buckets. I don't know that I have all the right buckets. But there's the sampling air, which responds to sample size, which is like historically, that is the science. And that still is going to be somehow there, although I care about the quality of the sample. But then there's just the modeling error, right? And there was just assumptions. And I mean, like someone went and redid the Wisconsin polls and they used different weights and they, hey, I get the right answer. Well, like they didn't know X-A-N-T what the right weight was. So there was just a range. And it's really hard to, to, to quantify that, right? But they're just as a source is that they, in this case, the percent of people who said they were very liberal was too high, and they really should have had a weight target for them to be lower. But X-A-N-T, we don't know that, and we could have been wrong in either direction, honestly. So there's the weighting. And then in addition, within all those subgroups, there's a non-ignorable non-response, which is just, you got to have the proportion right of whatever the group is. And then within the group, whatever answers they gave us, they could be unrepresentive of the whole population. So there's at least three sources. And then I think this is more, I think the active pollsters are also worried about just bots and authentic responses, which maybe that just folds into some version of non-ignorable non-response. But, but there's really a lot of sources for this. And only one of them is something that we can scientifically measure under very limited circumstances. We can and should offer people scenarios that are apart from statistics that are real world political scenarios when talking about possible outcomes. So it's not to say, "Oh, there's a margin of error in hey, throw up your hands. I don't know. It's somewhere in here." But you say, "Okay." The weights that you describe changing and getting a different answer means that if a certain group of folks do show up, even though right now you don't think they will, there's going to be a different outcome. Okay, that's a political story, and it's a real-world human behavior story, which is part of why we do this. Okay, and I've done this in the past. If young people do turn out, even though they say maybe now they're less likely, then this party is going to do better. And that's a way of taking the uncertainty and putting it into human terms, into political terms, which is a larger story than I think the public will have an easier time with. And that's just an essential part of the role. Well, we have about 30 more questions, but we are going to wrap it up with our final question that we always end with, and that is our magic wand question. And then I'm going to start with you here. If you could wave your magic wand, how much in the United States could any single candidate spend on their campaign? And that includes money raised, their own money, pack money, everything. What's that one number that they could spend? Well, from a polling perspective, I tend to think of money spent as what are the effects of advertising? And if you look around the world to all of us, whether we're getting products or watching something, seeing a show, et cetera, how much effect does advertising have? And what does it take to reach us? And there's obviously so many creative ways nowadays. It's not just seeing a commercial for this, it's targeting folks, building a profile, consumer profile of someone, pushing them something to, you know, their interest. I get a lot of snowboarding emails and ads because they know I like to snowboard. There's all of that out there going on. So what I don't know what the number is for how much someone could or should spend, except that I am always interested in the dynamic that happens when people are either trying to be persuaded off to one side of an issue or another. That is fundamentally why we do what we do. Or what public opinion does and how much it shifts, given certain variables and given certain inputs. So as long as that debate, discussion continues, then that is what it's all about. It's not so much the number as it is the dynamic. Michael, what do you think of? Yeah, so it's funny, especially early in my career, I spent a lot of time thinking about campaign finance. In general, the raising of the money can be problematic, obviously, if it moves you towards where the contributors are, but I think the trick of a good politician is to figure out how to raise money in a place where you are also very popular. I think we saw that, by the way, where that didn't work, was there were some Democrats in the primary season who saw a lot of money coming from APAC and then they raised a lot of money. That money was a negative, that negative, I think, clearly. That also then highlights, I think, to say that money determines elections. It matters and it can matter in certain contexts and not in others or whatever. But if money determines elections, we'd have President Jeb Bush. We wouldn't have President Donald Trump and others. And again and again and again, we have wealthy self-funded people just go down in flames. I don't think it decides elections, even though I also do appreciate that there are legitimate concerns in there. But in my answer is going to be, I actually think I would instead of, I think is it perhaps your question was phrased in terms of maximum or maybe it wasn't. I would just say a minimum that every candidate gets to spend $10 per constituent on whatever it is, advertising or staff or whatever. And then that actually then, that means in every district, both sides get to tell their story. And I think, honestly, you're a Democrat and you're in a heavily white rural Mississippi area. And he's going to win you that election, right? But still, he gets to tell your story and I don't think that's bad for democracy. And then in some of these competitive places, honestly, the big winners are just the candidates themselves. I don't have to spend all their time in a cubicle making calls, right? But I actually almost would define this more as a minimum and a maximum. Thank you all for listening. And if you'd like to continue the conversation in person, Anthony Savanto and Michael Bailey will both be speaking at the Political Analytics conference 2026, taking place October 8th and 9th in Washington, DC. To stay updated with all things, HDSR, you can visit our website at hdsr.mitpress.mit.edu or follow us on Twitter and Instagram at the HDSR. A special thanks to our executive producer, Rebecca McCloud and producers, Tina Toby Mack and Erin Keeswater. If you liked this episode, please leave us a review on Spotify, Apple, or wherever you get your podcasts. This has been the Harvard Data Science Review, Everything Data Science and Data Science for everyone.

Podcast Summary

Key Points:

  1. Election night at CBS News involves real-time analysis of exit polls and precinct-level data to identify consistent patterns before making projections.
  2. Polling is increasingly a modeling exercise, where pollsters use historical turnout data and sub-group analyses to predict outcomes, especially for hard-to-reach populations like young voters and Hispanics.
  3. Traditional polling methods face significant challenges due to low response rates, non-response bias, and shifting voter behavior, particularly in key demographics.
  4. Modern pollsters use voter history, turnout scores, and panel data to model electorate behavior, recognizing that polls are dynamic and responsive to campaign efforts.
  5. Prediction markets complement polling by offering real-time, instantaneous insights into public sentiment and confidence, especially on shifting electoral dynamics.
  6. Polling margins of error are often misinterpreted; they are calculated under idealized assumptions and fail to capture real-world complexities like non-response bias and modeling errors.
  7. The most significant sources of uncertainty in polling include sampling bias, non-ignorable non-response, and flawed weighting assumptions—especially in critical subgroups.
  8. The future of polling lies not in a single magic number for spending, but in understanding the dynamic influence of advertising, messaging, and voter behavior on public opinion.

Summary:

This podcast explores the evolving landscape of election polling, highlighting both its strengths and persistent challenges. Hosted by Harvard Data Science Review’s Shally Mangini, it features leading experts Anthony Salvanto of CBS News and Michael Bailey, author of *Polling at a Crossroads*, who discuss how modern polling operates as dynamic modeling rather than static surveys. Key insights include the critical role of turnout data, the difficulty in accurately reaching and representing marginalized voter groups like young people and Hispanics, and the limitations of traditional margin of error calculations.

Polling accuracy is now understood as dependent on complex models that account for historical behavior, non-response bias, and shifting voter intentions. The conversation also examines how technology—such as online panels and prediction markets—enhances real-time data collection, with markets offering complementary insights into public confidence and rapid shifts. Ultimately, the discussion emphasizes that polling is not about absolute precision but about capturing the dynamic, human nature of political opinion.

The episode concludes with a focus on democratic equity, suggesting that every voter should have a fair opportunity to be heard, even as campaign spending and messaging influence outcomes. The insights are framed as essential for understanding elections and maintaining public trust in democratic processes.

FAQs

Early in the evening, the decision desk starts with exit polls to gauge voter dynamics. Analysts look for consistent patterns across counties and precincts, using models that compare current vote trends to past votes. They also monitor differences between vote-by-mail and in-person voting, which often favor different parties.

Pollsters build turnout models using historical voter data to predict who is likely to vote. They assign turnout scores to individuals in their sample and adjust for groups with low response rates, such as young voters or Hispanics. These models help predict how turnout shifts can affect election outcomes.

These groups have very low response rates—sometimes less than 0.5%—making it hard to capture their true preferences. Pollsters struggle to reach and accurately represent them, leading to significant modeling errors and potential mispredictions in close races.

Key challenges include non-response bias, changing voter behavior, and outdated sampling methods. Polls often rely on assumptions that don’t reflect real-world dynamics, especially when turnout and voting patterns shift unexpectedly between elections.

Prediction markets provide real-time reactions to new information and can respond faster than polls. They offer insights into how traders perceive risks and outcomes, especially on downstream effects like control of Congress, and can serve as a complement to polling data.

The traditional margin of error assumes perfect random sampling and 100% response rates, which rarely happens. In reality, data quality, non-response bias, and modeling errors create significant uncertainty that isn’t fully captured by standard statistical calculations.

Chat with AI

Loading...

Pro features

Go deeper with this episode

Unlock creator-grade tools that turn any transcript into show notes and subtitle files.