An Interview with Jon Krosnick on Survey Design, Public Opinion, and Scientific Rigor

Welcome to the podcast. Total Survey Design. Welcome to season 2. Hello everyone. Welcome back to the Total Survey Design podcast. My name is Dr. Azdren Soma. In this episode, Sunny Up and I had the privilege of interviewing Dr. John Krosnick. We recorded our conversation at the historic St. Louis Union Station Hotel during the 2025 annual American Association for Public Opinion Research Conference or APOR 2025. Just to share a bit more about who Dr. Krosnick is. Dr. Krosnick is a professor of political science, communication, and psychology, and the director of the political psychology research group at Stanford University. This research group explores topics such as attitude formation, political behavior, survey research, and public opinion, particularly on environmental and climate change issues. He also advises the US Census Bureau, has allied national studies on voting behavior and climate change, and has published extensively both in survey methodology and social psychology. I also want to point out something relevant to survey methods. Among many other awards, he has won the APOR Lifetime Achievement Award for Outstanding Research. In our conversation, Dr. Krosnick reflects on his path into survey design, the role and importance of mentorship, and we spend some time discussing his work on measuring trust and improving validity in surveys. He discusses the founding of a new research center at Stanford and the deeper purpose of scientific work. We also explore the broader challenges of academia, including the publisher-perish culture and the need of a compassionate form of constructive criticism, as well as the value of humility, replication, and applying rigorous methods to real-world problems. It was an honor to chat with him. Let's dive into the conversation. When did you discover surveys, or I don't know if I want to phrase it as a love for survey, do you love surveys? Sure, I love surveys. Survey design? Absolutely. When did you discover that? I went to college having loved science and math in high school. I went to high school long enough ago that at least at high school I went to, there was no social science. I didn't even know there was such a thing, but I knew I liked science as a process. My dad was a medical doctor, and so I thought I was going to grow up to be a physician, because that was kind of all I could really, I didn't know much about professions truthfully, but I thought that was the likely direction. I had been a really serious percussionist and drummer in my early years, and so I did for a while think about, do I want to go the kind of medicine-science route or the music route? It became clear that I was good, but not that good as a musician, so I could better to do something else. When I went to Harvard as an undergraduate, for no good reason I declared myself a biochemistry major. You don't even have to declare a major when you get there, but I just said that anyway. Just because biology or chemistry seemed too simple, like I had something more, little did I know, biochemistry is really challenging. First semester at Harvard, we took four courses a semester, and so there was a required math course I had to take, and a required chemistry course I had to take, and a required writing course, and then I had a fourth course slot open. At the time, Harvard had a book that was about this big and two inches thick that described in detail every single course being taught at the university. At Stanford, we have this ridiculous rule that professors can only describe courses in 17 words, and they can't be complete sentences. It has to be like a little summary or whatever. But Harvard faculty wrote paragraphs, and so I don't ask me how, but I went through this whole book and I read the Psychology of Politics course description, and I thought, that looks interesting. So I go to the classroom, which is in the basement of William James Hall, where the psychology department is, and it's like 400 people, big classroom, and I'm just new at Harvard, I don't know what's going on. I'm sitting in the last row, I'm listening to this lecture, and it sounds interesting, and then at the end of it, Tom Pettigrew, who was the professor, said, "Okay, we're done for today. I want to see the following people up here now before you leave." Bing, bing, bing, Crosnek, bing, bing, what, what? So, okay, so I go up there and he says, "Okay, you're all freshmen. You don't have the background for this class. I really strongly advise you not to take it. But if you want to take it, it's up to you. Good luck with it to you." And so I ended up taking it and just entirely fell in love with the scientific study of human behavior and political behavior in particular. And Tom had always done survey research. He's very well known. He's alive still. He was very known. He's like really old, but he had always done research on racism and intergroup relations, which the Stanford psychology, social psychology program had a history of focus on. Gordon Alport, who was a gigantic name in that area, was at Harvard previously. And so, okay, so I was bit by the bug and then found a guy named Chick Judd, Charles Judd, who was at the time an untenured assistant professor in the psych department and took courses with him. And just I learned enough about him to say, "I want to grow up to be that guy." So I forgot, you know, he wrote a biochemistry switch over to psychology. And he eventually, I mean, I took his courses, other courses. And it was at a time when psychology was introducing sophisticated statistics in a way that it hadn't had before. And I took well to that. So I took graduate courses in statistics. And at some point he got an NSF grant and he needed a research assistant. And so he hired me as his research assistant. And the project involved survey data, here I'm finally getting around to answering your question, that involved national survey data, the American National Election Study data, that the University of Michigan had been collecting for a very long time. And the project really happened because of another guy, Mike Milburn. Mike, for a long time, has been a professor of psychology at the University of Massachusetts in Boston. And he had an idea about some of Phil Converse's very important political science research, where there was kind of a different way to analyze the data with new and analytic techniques, and that seemed to indicate different conclusions were merited about how the American public thought about these things. So bringing survey data together with statistics was when I really got engaged. And, you know, I had to like go, I don't know if you know the ANES, but the back in those days, there was a big code book for each year of the survey. And you had to read your way through this code book in order to understand how the survey was done. And so I, as an undergrad, I did that and, you know, got into the data. And that was that that was the start. So I mean, as an undergrad, I didn't do any type of like research work. So is that common at Harvard? I don't think so. I mean, there's kind of like two answers to that question. Answer number one is that for Tom Pettigrew's class, I had to do a class project. And it involved the survey, but it was also an experiment. So what what this, it's really kind of amazing that this happened, but I was sort of interested maybe in a career in the record industry, because my, you know, the music part of me coming together with, and I was really interested in jazz. So a friend of mine introduced me to a friend of his who worked for Columbia Records. And you guys may not know, I mean, Columbia Records was like the big dog label. And they had like all the major jazz artists who was I was interested in at the time were on Columbia. And so this guy was very nice. And I said, you know, I'm kind of interested in what your career is like. Can I come spend a day with you in the office? So I go spend a day with him in the office and, you know, kind of shadowed him. And at some point he said, you know, I wonder if you might like to help me with a research project that I'd like to do. And so it turns out that the way Columbia and basically every all record labels at the time would advertise a record was they would take out ads in magazines advertising the new Billy Cobham record to jazz fans in a jazz magazine, or maybe in Billboard magazine, which was a more general magazine of music. But they were always advertising one record at a time to people who they already knew were interested in it. And this guy had a really interesting idea. He said, you know, what about advertising the entire genre of jazz to people who aren't in the jazz, you know, aren't interested in jazz. They don't know they're not. So we made radio ads that had lots of snippets of jazz with a narrator and talking to people through it. And we bought airtime on Boston radio stations and actually, you know, did a field experiment. It was a, you know, what you call an interrupted time series in a way where we're like measuring sales of jazz records. Then we put the ads on and then we measured sales of jazz records after that. And as far as we can tell, they didn't have any effect at all. But I really love those ads. And we actually, you know, like 40 years later, I did an experiment recently where I exposed people in an online survey to the ads. And it turns out that they had statistically significant improvements in interested in hearing more jazz, interested in buying jazz recordings, you know, lots of dependent variables. So that, so one answer to your question was like in a class, I was, I had an assignment and so I ended up doing that project. Okay, great. And then, you know, Columbia gave me a bunch of money to do it, which was great. Okay. So that was, that's one answer. Another answer is that I think it almost never probably an undergrad would be a paid research assistant on an NSF grant. That was, that was incredibly lucky for me to have that opportunity. It's difficult to replicate the same experience. But I also, I mean, in my time in graduate school and as an undergrad, you know, the things that I value the most are the good mentors that I have. I think you would agree if I were to say something on the lines of like, you know, professors should like focus on, on, on being better mentors. But I think you're making a really important point that that for me personally, I'm incredibly pro mentor. I mean, I think it just makes life so much easier for all of us to see examples of folks doing what we want to do to be able to know how to do it and to know that we want to do it. And it's actually kind of interesting that I was talking to my wife recently and I kind of realized that she never had a mentor in her life. And she said that's true, that she was nobody ever who she looked at that way. But for me, it's like every time something good happens to me, I'm like, thank you mentors, you know, that like, those are the people who sent me up to have it happen. So I'm with you on that. Yeah, it sounds like you just had like, when you were describing the first few years of your undergrad career, let alone you just had so many good role models. To just, you know, take after and just model yourself on. And then I guess as a single, sometimes you eventually go a different direction from them or end up outgoing them or, you know, be levels with them. So yeah, I was going to say, I think it's real, just to add a little bit because you worked on the NES data. I thought it was interesting that you pointed out today, some issues with like, current measurements on trust. Yeah. And I mean, because I remember, so we took a political science class. And I mean, we did a whole week on trust and I know that we've, we read papers that use that data. So I think I thought that was really fascinating. And I was like, I was on his name. Yeah, exactly. Yeah, he's one of the big dogs in that area for sure. And you know, there are people who love that dichotomous item that are, you know, it's just been used a lot. They've generated a lot with it. But in our case, we felt like, can we do better? And then it turns out we could do better. But I think you also point out some really good points. And I mean, we've worked for the social and economic science research center as a survey design consultant since like, one of the easiest things to point out in a survey is like when a question is double barreled. And like, that's one of those questions where I'm like, oh, wow, like, what are they measuring? There are two things that are measuring. So just the Q and the audience, could you just tell us just a little bit about that question? The original A&ES question, measuring trust? Yeah. So the A&ES doesn't get credit for creating that question. It was created earlier. And yet, you know, the general social survey and the American National Electric Study both adopted that question as the standard. And the question says something like, do you think most people can be trusted? Or do you think you can't be too careful in dealing with other people? And what I talked through today in the presentation, which is research that I've done with Sebastian Lundmark at the University of Gothenburg in Sweden, is that the question is kind of wacky right there. As you said, I mean, double barreled is kind of a nice way to put it, that it's one asking about whether most people, it's kind of like a yes/no question, most people can be trusted. But then it's got this other side of it of being careful, but not being clear exactly careful how. But when it's in the context of the first part of the question, at least you kind of realize that it means being careful not to be taken advantage or by somebody who can't be trusted. But, you know, the problems that we identified were in addition to that, the fact that any dichotomous item like that, there's almost certainly a continuum. There are kind of more points on the scale than just two. And so we did find that, you know, adding in experiments that we did in national surveys, that by adding more points to a rating scale, you got more validity. And interestingly, when you try to optimize the design of it, we thought about, well, you could make a continuum in terms of how many people can be trusted. You can make a continuum of how often people can be trusted. You can make a continuum of the probability that somebody can be trusted. And it's like, I don't know of a basis in theory to prefer one over another. They're all kind of plausible ways of doing it. And as I showed in the paper today, do any of them is better than doing the dichotomous item? Can I follow up just a little bit on something? So you're talking about figuring out a better measure with greater validity. And this is something that I'm still learning about. So I'm just curious, like, what are the things that you look for that show you something is more valid? Like a certain survey question, right? So I think this would be helpful even in other survey measures and other survey questions. So like, how do you know a question is valid? Like, how do you know one question is better than another one? Like, what do you look for to prove that internal validity? So we're external. Right. Okay, so let's sort that out for a second. So internal and external validity are actually attributes of a study, not attributes of a measure. So internal validity refers to, if I do a study and I want to argue to you that this advertisement made people more favorable toward jazz, how confident can you be that it's really the causal effect? That's what internal validity is. And so, for example, I could completely ruin the internal validity of an experiment by having the treatment group not only listen to my ad for jazz, but also listen to an ad for football. And now you don't know what's causing the difference between the treatment and the control group because there's a confound in the design. So that compromises internal validity. So that's what that's like within a study. External validity is about the degree to which the findings of a study generalize to the world. So if I have online survey experiment people listen to my jazz ad, seems to make them more positive. How confident can I be that if I did that in the real world with real people and radio, would it really work? So you're asking, though, about measurement validity. And the American Psychological Association was very helpful decades ago in talking about different styles or types of validity of measures. And what I talked about today is sometimes called predictive validity and sometimes called correlational validity. So let me walk you through this. So I think your question is right on like you're saying, well, okay, you said that this measure of trust is better than that measure of trust. How do you know which one is the better one? And so the answer to that is that we can't know because like in a paper that was presented at APOR this afternoon, we did something very different. We were asking, do you have diabetes? Do you smoke cigarettes? Things like that where you actually can look at some external criteria and compare and say, is this measure accurately reflected? In this case, if you ask somebody's opinion or how much they trust, well, there is no truth of how to measure trust. So what we do is the following. This is the way predictability works. That you have some people answer the two point, the dichotomous question and you have other people answer the five point rating scale and everybody answers some criterion questions. So what I said during my talk today was that the criterion questions are measuring things that theory says should be related to trust. So one example is voting. So there's a literature that says people who are more trusting of others are more likely to vote in elections. So you ask all the participants in this experiment, did you vote or not? Then you do a regression where you predict voter turnout with this measure of trust and this measure of trust. And if this is a better measure, it should predict more strongly. And those are the findings that we showed in that paper today. So that's only one. Another way to think about it is let's just say that we're going to do an experiment. And we have done this experiment. I'll describe something real to you. There's a research tool called Cyberball that is a very, very interesting, highly, highly used, basically video game, very simple video game, where you bring people into a lab and you have them look at a computer screen. And there are two little stick figures on the screen, a little circle head and a line for the body. And one of them is holding a ball. And the idea is that you're going to play a game of toss the ball with these stick figures, like you're the third person in it. So each time somebody gets a ball, they throw the ball. And so like when the stick figure on the right throws the ball, they could throw it to the other stick figure or to you. And then when you get the ball, you could throw it to that stick figure or the stick figure. And so what you just basically, just like little kids on a playground in school, you're throwing the ball back and forth. What the experiment manipulates is how often they throw the ball to you. And so what they do is they create what's called ostracism. So look, if they throw the ball back and forth to each other and you don't get it, it's amazing how much that feels like the playground. Like playing that game, it feels really unpleasant, even though you know you're playing with two stick figures in a computer, but you feel excluded. And it turns out that reduces trust, generalized trust in other people. If you play that game and you get ostracized. So now the question is, OK, if we have two measures of trust, let's see whether the effect of the treatment on this measure is stronger than the effect of the treatment on that measure. And if that measure is the more valid one, the effect of the treatment will be stronger on the more valid one. Because you're getting rid of noise and you're picking up the signal that theory says you should expect to see there. And so you've got theory on your hand, on your side on the one hand, and then you've also got the evidence of the effect happening on the other hand. So lots of studies like that are offering now what's called convergent validity. So that is like a series of different studies using different methods that all support the idea that the five point scale is better than the dichotomous item. And that's how we do it, because we can't do what Elizabeth Elder talked about in her papers this afternoon, which is, you know, compared to some measure of truth, because there is no measure of truth. I just want to say that's fascinating. No, the cyberpolo experiment, like it's so interesting because as sociologists, I've always loved George Zimmel's idea that triads are the smallest unit of societies. So yeah, like when three people including yourself are doing this really no rule ball tossing game and you're left out. Yeah, that means something that invokes some emotion, some effect, and then again, create some values inside some values of ideas of trust. And yeah, so it's like really interesting because I've done experimental studies. You have done experimental studies in the past. Yeah, so then to use that the game with the ball, you have to bring people into the lab, right? No, we had people play cyberball from their homes in online service. Oh, wow. Yeah, I just imagine like it has to be like a software that we install, but you know, you don't. I mean, in the old days, so this was invented by a guy named Kip Williams, and Kip initially was doing it in labs and had people come in. But now you can run it online with no special software just in the context of an online survey. That's pretty cool. I need to look that up. Yeah. No, I think. Yeah, cyberball. I appreciate sharing that story about that. So I was at your session about the people from different universities talking about how and how like how they came about and how they are running their survey research or institutions or organizations. They have their current institutions. But, well, first off, congrats on your launch. Yeah. But I just wanted to hear what were the inceptions of that institute, like what brought about that institute. And, you know, maybe you're kind of a personal anecdote that led up to that start. Yeah. Well, there's kind of two answers to it. The way I think about it, I spent 18 years on the faculty at Ohio State when I came out of grad school. I was a professor of social psychology and political science there. And we had a great time in Columbus. Our daughter was born there. Raised her up into middle school. And I have a second life as a jazz drummer, a professional jazz drummer, and it was a good place to play music. Ohio was great. But after maybe 12 years or so, I qualified for a sabbatical. And all of my mentors had sabbaticaled at Stanford at the Institute for Advanced Study there. That's the CASBUS. It's called the Center for Advanced Study in the Behavioral Sciences. And it's an amazing place. It's up on the top of a hill overlooking the campus. There's 50 offices. There's a dining room. There's meeting rooms. And you go there for a year and you just work in your office and then you come out and have lunch with all these people. And then you enjoy California. And so we were there like two months and I realized I would like to live here. So I started looking around for jobs, but there weren't really faculty jobs. And so we went home. And then a few years later, Stanford called and said, "Yeah, how about you come here and join the faculty?" So after we moved to Stanford, Harvard realized that we were movable. My wife is a professor also. And so they said, "Well, maybe you'd like to come here." And so we said, "Well, we'll interview." So we went and did interviews. And as they were offering us jobs, the president's office basically did the recruiting of senior faculty there. The woman in that office said to me, "Okay, we want you to do something really big when you come here. Like, tell us something, you know, it's a huge project and we'll fund it and then you'll do it." And this was, you know, it was an important way for them to recruit faculty. And I think they, it's funny, I mean, my wife and I had both been undergrads at Harvard. So you would think maybe we would be attracted, but it was kind of the opposite. Like, we felt a little bit like, been there, done that. Didn't they go there again? And California was new and fun. And so, okay. But so I said, "Well, here's something I could do. I could create a Center for Excellence in Survey Research that is envisioned exactly the way you heard it yesterday." And so they said, "Good, okay, we'll give you the money for that." And so I, if I had gone to Harvard, I would have done this 20 years ago. But as it turns out, we were in love with California and Stanford said that they would support it, but it took this long for it actually to happen. And so, and it's really importantly due to a guy named Mike Tombs, who is the chair of political science. I'm actually on sabbatical this year, so I haven't been on duty, but he's been creating this Center and launching it. And so it's really, it's really exciting for us to have the opportunity to hire staff and to launch down this pathway of creating what I call the ball of knowledge, which is to pull together everything we know from lots of literatures on best practices in the conduct of surveys, and then to use the ball of knowledge to give the profession more of a leg up on best practices. Because if you just think about Don Dillman, who you worked with, the guy produced a boatload of books and articles and all that kind of stuff. But if you went down here to any arbitrary person at APOR and said, "How many of those articles and books have you read?" "No, I don't know a few." But then there's also Stanley Presser, and then there's also lots and lots of other people, Nora K. Schaefer, who got the award last night. It's a gigantic pile of literature. You cannot do that as an individual. You certainly can't do it as a student. Somebody kind of has to do it for you and distill it and has to be on duty, pulling that literature together, updating it in real time as new stuff gets published, and then helping you as a practitioner to learn from it and do your work well. And so that's what we're hoping to do. First of all, thank you for bringing up Don Dillman. One thing that I do want to emphasize about Don that he always promoted was good sciences, science that has communicated well. So things that actually reach the audiences that need it. If anybody looks at his books, they're always very easy to digest. He summarizes the points really clearly and then goes over them very clearly, point by point. So a lot of the feedback that he gave me on my dissertation was about asking the questions like, "Who are you really trying to help? What's the ultimate goal of your research? Is this really important?" And just thinking about the big picture and thinking about, "Okay, let's be practical here." Sure, there are a lot of details that are important in dissertation, but ultimately, what's the goal or what is the ultimate outcome of this? Another thing that he just said, which is really important, is I heard a lot of names yesterday that I don't know. And I feel like I've been so invested in the survey methodology for the last few years that I'm a little disappointed with myself because I'm like, "Who are these people that are really highly praised?" And I thought I was actually doing pretty well for myself as a survey methodologist. But so I do think it's really important to have somebody who can curate that kind of a library and that kind of knowledge. And yeah, so I'm looking forward to seeing how that turns out, how your project at Stanford turns out. Yeah, I mean, this is my first time attending both of us. And now I'm at an institutional research office. I got a lot of flack for it yesterday, which is valid. Which is for me, which is valid. But I mean, I also have to do a lot of due diligence to, you know, do that battle. But being the only survey or self-claimed and also, you know, publicly now, survey methodologist, but the only person in the office, I feel like I was doing this lonely battle, lonely work. But coming here, I feel like I got just a lot of moral support and camaraderie. And I feel like, oh, tomorrow, when I go back to Kansas, I feel like I have this energy to fight back. And maybe having put faces through those names, I might be able to, you know, maybe have the courage to ask them, like, hey, like, I'm stuck with this issue. Like, how would you have, like, you know, navigated this kind of, not just methodological issues, but actually, like, you know, talking to higher ups, like, no, we actually have to change the way of wording the questions. And why not? And how do we even approach the higher admin folks, those kind of things. It was a very, very encouraging experience for me. And imagine having, you know, geographically in the same office, having people working together on like, same projects. That would be just a very awesome thing to have. Yeah. And you may have that someday. But, sorry, whatever you're about to go on to, I want to just resonate with something you said earlier before we do that. So don't forget what you were about to say. I think you raised something really important about Don Delman. I think when Don told you, think about the effect you want to have and is it important and why is it important? It took me a really, I wish he had told me that a long time ago. It took me a really long time to learn that lesson. Because as an assistant professor in psychology, I just thought I got to publish in my discipline to get tenure. And I was only thinking about other psychologists, like the people who are going to write the evaluations of me and the people in my department. I wasn't thinking about the world. And when I spent that year at CASBUS at the Center for Advanced Study at Stanford, the thing about it is, out of the 50 people, there's like three psychologists, four sociologists, three economists, and then, you know, a bunch of people from lots of different fields. And when you give your seminar to that whole group, you can't do it the way you would have to just your psychology colleagues. You're speaking to a larger audience. And I was like completely at a loss. I really didn't know how to do this and I had to figure it out. And when you realize you got to think about why is your work important to the world? It's just a whole, at least for me at that time, it was a whole different question now. Admittedly, social psychology is very different right now. Social psychology came to realize that being relevant to the world is critical. So there's tons of studies on racism and other really obvious social issues. But back then, we were like in the mind and doing basic science studies of how influence happened without thinking about solving world problems. And so Don would have said to me, "John, think about what's the effect here?" And so you're lucky to have had that mentorship. Like, I remember as it came back after this one meeting, Don totally just tore it apart. Like, what is this thing? Like, why are you even having this survey? Like, you're not thinking about the broader public and he came through devastated. But thinking back, maybe he was right because it wasn't just getting this published or getting the survey done and not getting the responses done. But yeah, there is way more. I have to elaborate on this story. I went to Don with my first version of the dissertation survey, the questionnaire. And he gets read and he looks at it and he's like, "Did you even, have you even done a survey before?" You took my class, but like, did you really even, like, I can't believe he started, he turned red. He was frustrated, but here's the thing, he didn't give up. Exactly. That's the point. He spent three hours that afternoon with me in his office working on it. Exactly. And, you know, he tore it apart, but he gave really good feedback, honest feedback. Yeah. And he really helped me improve it. I mean, yes, it was hard. But I mean, I think the thing for you to never forget is that he would not have spent that three hours with you unless he had confidence that it was worth his time, that you had the potential to gain from that and to carry it forward. And so as much as it's a huge loss for everybody to lose him, you're in a very special place to honor his memory and to make him proud, you know, by what you do. And I really encourage you to do that because, you know, it really, you got an investment from a world-class mentor. And, you know, that, I mean, that's why you're sitting at this table right now. And so I think that's, you know, that's a really positive statement about you. And frankly, I can feel this about both of you already. I can feel that you have the potential to be really, I mean, you just, everything you're saying is like, to me, signals of like fabulous professional in the making. And so, you know, that, like, you're, that was very sophisticated, which you said about not only needing to know how to do methodology well, but you got to sell it to the people who have to be convinced to let you do it. And you're right that there's a whole, you know, there's like dozens or hundreds of people who are here who have some advice for you on that. And the great thing about APOR is it's so mentor-oriented. It's so that people are so willing to talk and, and, and help young people coming up. And, you know, yes, you guys are young relative to us. So, you know, thank you. I mean, this ties back to the mentorship bit because, well, I mean, when we're just trapped in this, you know, like, publish or perish or mantra or just, you know, getting the work done or just bureaucratic nature of mentorship and grad school. It's so easy to fall to the allure of giving false positive reinforcement. Like, no, like, you're, you're, like, your survey looks fine, good enough, just get it done, leave the program and we won't see you again. But he, like, I think that's one critical part of good mentorship is like, you actually do not give false positive reinforcement. And I mean, it's painful because it takes, it's difficult to bring up your, your work sucks, but also you have to take that time to actually tell them why this sucks and what, how this could be improved. So it's, it's difficult job on both ends, but it's, I think, an effort to create a better good. Yeah, it's not as hard as you think because for us in our position teaching you that we actually, you can think of it as like self, selfless. Are we willing to help you be generous with our time and our knowledge, but it's actually also selfish for us because if we mentor you and advise you and we have confidence that you're going to use our advice and you're going to go off and do better work as a result of it. We will take credit for all your success later and then we will smile and we'll, you know, we'll, we'll say, I helped him. I know, you know, I, that's, I'm very proud of what he's doing. That, there's nothing makes people in my position happier than having that experience. And frankly, that's why I think teaching is addicting that once you, once you put energy into it and you have, I mean, I'm amazed walking around a poor, how many of my former students are here, you know, and, and like, oh, he's doing Google. Oh, she's doing, you know, I mean, the editor of public opinion quarterly, the journal is my former student. And, you know, so like there, it's just, it's very rewarding. So it's not, it's not as selfless as you might think. But I have to add something. There's something to say. And I think maybe I'm wrong, but I think it's a cultural thing where, and this is my own personal experience. I think a lot of academics that I've encountered have a difficulty in giving that harsh criticism. But on the other hand, I'm like, I understand that for good research, and I think it's healthy for academia to have that kind of like harsh, but objective or critical feedback. And I was a little bit disappointed. It was like, with a lot of, you know, my professors that they didn't give more harsh feedback. And I think, you know, I was thinking to myself, I'm like, okay, maybe Don was in a position where like, I met him and he's like, 45, 50 years into his career. And he just doesn't care if somebody thinks that he's being too harsh or something like that. Like he has the authority and he's able to do that. But on the other hand, I wish more professors were more direct and more critical, but like critical in a constructive way. Yeah, so. Yeah, I can tell you that is hard on our side is how to balance giving useful criticism. I mean, step one is that if you whatever questionnaire you walked into that office to show him, you have to be able to tell the difference between that questionnaire and you as a person. You are not that questionnaire. And, you know, it's you want him to say, wow, this is fabulous. But the truth of the matter is you more want him to be honest with you and to help you improve as a professional. And so that I think you're being very mature about that. I don't I think a lot of people, you know, just like, well, just be nice. Just tell me it's, you know, good. And that when you when you give negative feedback to students, what I've learned is that you have to calibrate it to what the student wants. That if the student just wants to be praised and get past this experience, let them do that because there's no point in trying to make people performing at a certain level that they're not motivated to do. On the other hand, when you see somebody who's really got the potential, you got to balance the feedback, the constructive feedback and the encouragement with what might seem to be critical and making it clear this is a criticism of the questionnaires, not a criticism of you. And the problem is that for many years after where you are right now, that's a really hard distinction to make. It's like if you get a paper rejected when you're a third year assistant professor, you think maybe I can't do this. And it reflects back on you. And that's a totally natural human thing, but it's not always good, you know, just to subject yourself to that kind of, you know. So there's a question that I had from earlier. Yes, please. Yes, that I wanted to ask. And this is like one thing, one part about your story that I am curious about. And I have to ask before this conversation ends. I'm really curious how you went from that person doing the research assistant for Capital Records to eventually getting so involved and interested in survey design that you go on to learn a lot of words here at APOR. And you publish a lot of papers and even books on survey design. So then, like, what sparked that difference that took you from a psychology student in Harvard to being this very well-known survey methodologist? Yeah, I mean, I would say first, it's a little bit of a complicated answer, but step one is you got to love the work. And Chick Judd, who really showed me, you know, his office was next to mine. I drove him crazy by going in his office all the time to ask questions and do things. And, you know, he really showed me how to be a researcher, how to think about a research project. And I just loved it. I just, you know, I didn't have anything like that other than music in my life where I had, I found a process. And I didn't understand at all what we were talking about five minutes ago. I didn't understand where this was going to go. Who was this going to affect? I just, like, trusted him, basically. And so I enjoyed that. Then I thought, okay, I want to do this. So I got to get a PhD. Where should I get a PhD? And he said, you should go to Michigan because Michigan has people who are on the faculty are really strong statistically who you study with. And who are, and there's great psychology there. So I apply. Luckily, I got into Michigan just as a little footnote of entertainment. My wife and I had started at Harvard the same year, but she took a year off. So she was graduating a year after me. So I got admitted to Michigan and I called up. I said, my, my, I mean, not even wife, like, you know, I have, my girlfriend is, has another year at Harvard. Could I delay my admissions for a year? And so they said, yes, you can, but you lose your funding. So I had like a fellowship that gave me a full year of money without having to do anything. But we'll find money for you somehow. Okay, fine. So I go and they did, they made me a TA first year. Fine. So I got through that finish. So I waited for her. She finished. We went to and are together. And I studied there with sociologists and psychologists and political scientists and was at this interface of survey methods and political science and social psychology. And, you know, so the, the answer to your question really was I worked with great people. They gave me, I said, what do you want me to work on? They gave me great projects to work on. I learned how to be a scientist. I learned how to love being a scientist. And, and I just did, I just was committed to, to doing good work by the standards of the science. And one of the features of me, I think that distinguishes me from some other people, but not done so much is that I, I, once I pick a topic and I study it, I don't let go. I want to keep investigating it. So one of my lines of research that I think has been the most rewarding in some ways has to do with election ballot design. So if you think about a ballot, it's kind of like a questionnaire. And when you walk into a voting booth or you get an absentee ballot, the candidate names are in a particular order on that ballot. And in surveys, we know that the order of answer choices influences the selections people make. So why wouldn't it happen in an election? Well, an election, are you kidding? I mean, this is so important. How could it matter what order the names would be in? But it turns out people vote in elections often without a strong preference. They're feeling ambivalent. They're not sure who they want to vote for or they don't know a lot about the candidates. And it turns out that order is a nudge. So what I found early on when I started to study this was that candidates get on average about 3% more votes by virtue of being first. So if you and I are running against each other for dog catcher and you're first on the ballot, you get three extra points. Well, those three points come from me. So now it's a six point gap. You're winning by six on average. And half the time it's even bigger than that. So a bunch of states like Ohio rotate candidate name order across voters to make it fair. But most states don't. And so, for example, in Florida, the law is that it's the party of the governor that gets to have their candidates first on every ballot. And so Republican governor in Florida, that means all the Republicans and all races throughout the whole state have this advantage by virtue. So we published a bunch of studies in this area showing that being first helps. I've testified in a number of court cases now with New Hampshire as an example started to rotate name order like Ohio that they hadn't been doing before. So that's an example of taking an effect that we survey researchers knew happens in questionnaires showing that it happens in this really important real world situation and then using the legal system to improve fairness. I mean, that's incredibly gratifying for me. And so that's because I kept doing more and more studies on it. And I just wasn't. So the first time I testified on this because, oh, your only evidence is from Ohio. Why should we believe that also happens in California or happen somewhere else? Okay, fine. So now we got to get data from California. All right. After we study data in California, say, yeah, that's only Ohio and California. But how do we know what happens in New Hampshire? Okay, fine. Study New Hampshire. So I'm willing to kind of stick with it and do the extra work that's needed instead of saying, oh, I already showed this on board with us. Let me move on to a new problem. So I think that is something that distinguishes me where the accumulation of the expertise I get from multiple studies in an area. But also, I mean, to be fair, we did a meta analysis recently. There are more than a thousand tests of this now. There were publications before us, but also lots of people got inspired to do this kind of work in various different countries. And so, you know, that the, I think the answer is that just like keep your head down and do the work kind of is the logic of it. And so, you know, I mean, why did I win awards? I have no idea. You know, I mean, but I mean, I think awards are great because they, you can look at that and you can say, okay, well, let me see what a model of a really successful professional. Not because there aren't 70 or 100 or 200 other people in the April organization who could have qualified just as much as I could. But you heard Nora K. Schaffer get celebrated last night, and that's an example of somebody. You didn't hear the half of it. You didn't, you know, that was not that presentation did not even begin to cover the richness of her career and her contributions. And but I thought it's really helpful for you to see that, right? And just to have them like maybe you'll go home and you'll read about Nora K. Schaffer and you'll figure out what her career was about. And so I think it's, you know, the bottom line is doing good work. But I think you said something important earlier that's relevant also that you got to be willing to communicate it to the public. You got to be willing to do public lectures, court testimony, testimony in Congress, that you have to be willing to communicate the work to a larger audience. And so, I mean, I've worked on projects with NASA, you know, major surveys of pilots, air traffic controllers. I've worked with lawyers who have gone after oil companies for oil spills and, you know, all kinds of opportunities to apply my scientific skills to bigger questions that are not just building the scientific literature in academic journals. And I think you got to be willing to do that, you know, but don't like not now, like later, you could, you know, worry about that stuff. But, you know, for now, just just do good work, you know, do it passionately, do it with the right principles. And as you are, I think, alluding to earlier, do it sensitively, navigating the waters of the people you work with so that you're promoting good science without pissing people off. Right. And, you know, you can do that. You can do that. So, like, one year ago, after I defended them, I had to clean out my office. So, I'm cleaning out my office, talking away, like, you know, old documents and turns out on a notebook, first page, it had 10 commandments of grad school by Don Delvin. And one of them was like, find a specialization. It's more like, and he said, like, hone in your quantitative skills. But, like, few points down, it was like, find a specialization. So, I think, like, that kind of resonates, like, on top of the methodology, you really found the public, like, the political ballots as, like, a subject matter and really were able to focus on it. But I recall you yesterday mentioned about, like, how, you know, not only surveys are used for political, like, research and academic research, but, like, litigation purposes too. Like, so, how has the new expansion of your career at the current stage, like, how, like, what are some new horizons or new fields that you have been, like, testing nowadays or recently? Well, my professional work is really not so forward-looking as your question. So, what I have is a huge pile of in-process research projects with lots of people. And I need to finish a lot of papers. So, I'm mostly focused on finishing those papers and books that are kind of in process. So, that's point number one. Point number two is that as far as institutional innovation, creating this institute for survey excellence at Stanford is going to be a big focus of mine when I return from sabbatical in the fall. And I'm not really looking to do a lot of things other than what I've been doing. So, I'm an employee of the Census Bureau. I'm going to keep working with the Census Bureau to help them do good work. I do expert testimony and litigation regularly when I get invitations to do that. I do it. So, my professional life has been fairly rich in having a variety of opportunities. And so, I'm not, you know, I'm not really looking to, like, bring new stuff in. If somebody calls me and says, "Would you like to do X?" I mean, I'll consider it for sure. But I'm happy enough just trying to finish up what I'm doing now and catching up basically that I'm not exactly looking for exciting new projects. I mean, so it's really interesting because some people, like at my career stage, are running, not walking toward AI. I have no interest in AI at all. I mean, I'm happy they're doing it, but I don't want to do it because I have my hands are full already with my stuff. And I don't think any of my stuff is not relevant anymore just because AI arrives. And so, you know, let them do it. And they'll tell me whatever they find. And dear God, I hope it all works out okay. But, you know, that, yeah. I mean, that kind of clicks because I don't know if that was a good observation on my part. But even from the presentation earlier, data that you were analyzing were necessarily like very, very new. It was like 20 from 27 to 18. So, I guess that kind of speaks to it. I mean, there is a little tendency, I think, in our field to be so sensitive to timeliness that 2017, why would we want to look at 2017 data? Because we're like, you know, but if you look at our ballot order stuff, the effect of candidate name order on ballots was shown in the 1950s. And now it's shown in 2020s, you know, and so this is something fundamental to humans. It's not like, oh, it's out of date, it's not going to happen anymore. So, I feel like the stuff I studied transcends time. Yeah, I was going to say, you know, good sociology is a sociology that, you know, should not be valid only in one particular point of time. You know, whether it's like, you know, the social psychology stuff of, you know, in-group, out-group queuing and that kind of stuff where I just, I do want to say, I want to be respectful of your time too. But what I wanted to ask as a wrap-up is kind of like give you the floor. This is a podcast on survey design and the audiences, I mean, we're assuming the audiences from amateur to like more professional survey designers. So, is there, if there's anything you want to add and say to the public of people interested in survey design? Yeah, I think that, I think the best advice I can give anybody in this field is humility. That I think there's, it's too tempting to think, a survey, what's hard about that, like a six-year-old could do it. They like write some questions out, what's your favorite ice cream and then they give it to like five people in their class in school and then they've done a survey. So, and that's true, that was a survey by some standards. But we know now that it's much closer to brain surgery than that. That there's huge literature on how to do every aspect of it well. And if you don't, if you, if you overly confidently think, I know how to do this. And as long as I exude confidence, other people will believe me too. That's a disservice to everybody, including yourself. And so, the humility, having the humility to realize that you need to learn what all those people at APOR already know. I mean, like just as you said, you know, you can't get it. I mean, APOR is like, whatever, four days, you have to come back to the next day, and the next day, and then find ways to interact with these people. Like watch every video you can of these people online, giving lectures. Take, you know, the University of Michigan survey program has Coursera courses. Take, do those on your weekends, you know, like, and get to know people so that you can use, as you said, use them as advisors and draw on their wisdom. And so, I think the more all of us are humble. I mean, this is the thing, you know, I described that we're creating this center at Stanford, and the first thing we want to do is create this ball of knowledge. And so, one of my colleagues at Stanford said, aren't you the ball of knowledge? I said, God, no, I'm not that, you know, there's like, I don't know any of that stuff. I can't, somebody has to be their job. I'm busy doing all kinds of other things. So, that, so I think that kind of humility is really critical so that we improve and so that we do the best work that we can. And I heard some presentations today that were absolutely the opposite of that. Like, you know, like, and so that, you know, that there's, there's that to deal with. But, you know, that's, you know, that's neither here nor there. But yeah, I would say, you know, that there is a lot of wisdom in the literature that is scientifically validated. But at the same time, I think it's important to recognize we haven't talked about the crisis of replicability in science. But, you know, in the last 10, 15 years, we have come to realize that a lot of stuff that got published in journals wasn't replicable for various reasons. And so it's really important not to overreact to one study, that you have to have evidence of replication in order to really have confidence. And so that's a pain, you know, because now you have to, it's more work to figure out, you know, if I just gave you an article and you say, well, believe that, okay, good, run with that. But now I say, no, no, you can't believe that you got to look for other other articles that have done the same thing. That's going to take more time. But I think we've learned we have to do it. And the standards of excellence have to be increased. And luckily, it doesn't seem like the world knows about this crisis too well. So we can get our act together before it's a public disaster. So I think that that's what I would say, you know, kind of as closing remarks for now. Well, no, well, thank you so much for your time, because I mean, we've just bumped shoulders so much so so many times past three or four days, but sitting down in your hotel room this like surreal almost. I really appreciate it. I mean, I learned a lot even just from this conversation. So yeah, thank you so much. Well, I mean, this conversation is more directed at you than like all the rest of the stuff you heard. Correct. A lot of people great. Yeah, like this is a priceless like topper to the first conference at a board. I couldn't ask for anything better. So I think you're I think you're I think you're wise to do what you're doing, right? You didn't just ask to have a chat, but you said you want to do a podcast. Why? Because you want to share this with other people and it's it's your it's an excuse to have a conversation with individuals like me. And then, you know, but like I said to people, I got to do this podcast and like what podcast what's going on there. And so then they find out what you're doing and that starts to create a buzz. So when you finally release this stuff, they oh, I heard there was a podcast and then, you know, so I think it's I think it's a really good idea and it's a way to get people to pay attention to you. And then frankly, I think you guys make great impressions. So as you talk to more and more people, you'll spread your own reputation. Oh, I talked to those guys, you know, and so that's a good thing. Well, thank you so much. That was Dr. John Krasnik. And as he said, it is our pleasure to be able to share our conversation with you. If you would like to learn more about Dr. Krasnik, you can visit his website at johnkrasnik.com. If you enjoyed this conversation, please consider subscribing, sharing this episode or leaving a positive review. If you would like to contact us, please email us at [email protected]. It's always fun hearing from fans of the podcast. And as always, thanks for listening. [Music]