Is a ton of psychology just ... wrong?

If your best finance people are doing expense reports, chasing receipts or spending time on month and close, it's time to get Brex A.F., agentic finance that eliminates that work before it starts. Learn more at Brex.com/A.F. Running a business shouldn't feel like surviving a software group project, one app for accounting another for inventory another for sales, and somehow none of them talk to each other. It's where Odo comes in, an all-in-one business management software that brings every part of your business together. From sales and accounting to inventory and marketing, all-in-one powerful platform, no messy integrations, no bouncing between tabs, and best of all, no spreadsheets. Stop managing software and start managing your business with one unified system. Try for free today at Odo.com/Vox, that's Odo.com/Vox. This is unexplainable. I'm Noam Hasenfeld here with our Senior Science Reporter, Brian Resnick. And Brian, as always, we're here to talk about an unanswered scientific question. Yeah, I actually have this really huge, kind of uncomfortable unanswered question. Okay. How much of psychology is wrong? Do we have reason to assume that tons of psychology is wrong? Yeah, so ten years ago, researchers started to notice and to really get worried about some really rotten things going on at the core of a lot of psychological research. It led them to really question and rewrite a lot of assumptions, and since then, there's just been this kind of huge fight. It's really a crisis. Very, okay, where do we start here? Let's start with the psychologist who's been working on this problem for years now. I'm Simeen Vizier, and I study how we can do psychology better. Simeen said this crisis really started getting going in 2011, when this one paper came out that had some really weird claims in it. Yeah, so it was a paper claiming that people have ESP, like predicting the future. Like ESP, like sci-fi, knowing what's in people's heads. Yeah. Okay. Yeah, this paper was from a psychologist named Darrell Ben, an interesting guy. And in it, he wrote up a lot of experiments. This is an experiment that tests for ESP. It takes about 20 minutes and is run completely by computer. This is the language participants read when they, you know, sat in front of a screen. On each trial of the experiment, pictures of two curtains will appear on the screen side by side. The computer would put a picture behind one of the curtains, and behind the other one would just be a blank wall. Your task is to click on the curtain that you feel has the picture behind it. If participants guess correctly, Ben took that as evidence that they predicted the future, that ESP is real. Yeah, but it's not real, right? Yeah, yeah, not according to, like, laws of physics. Okay, according to those things, sure. Yeah, no. And for most of us, we thought that the claims were preposterous. Okay. Proposterous. I mean, is that just a nice way of saying this is all nonsense? Oh, it was nonsense. Okay. The results here were absurd, but this is the thing. The methods Ben used, those weren't absurd. It was a nine-study paper very methodologically rigorous by the standards of the day. If this paper had more plausible results, like if you sub in ESP for, like, something more normal, it would have been viewed as a great paper. Okay, but if you're using all the right methods that psychologists everywhere agree are fine and legit, how do you get a result that's as absurd as people can predict the future? No, this is the key question. So shortly after this whole ESP paper comes out, another paper comes out. So it's a 2011 paper by Simmons Nelson and Simonson called False Positive Psychology. And it really answers your question. It shows really clearly how absurd conclusions can come from seemingly like well-built studies. The basic idea here is experiments generate a lot of data, and a lot of it just seems like noise. But you could do a thing where if you only publish the patterns that are interesting to you, like within that noise, like you can get these results. Let's say you tell me that you accurately predicted the winner of the Australian Open. You know, two days before the Open, before any of the players get on any of the courts, you wrote down your prediction on a piece of paper. And you told me that Naomi Osaka, she's definitely going to win this. And then lo and behold, Osaka is champion in Australia for the second time. If I know that's the only prediction you made, I'm going to be much more impressed than if you had 50 other pieces of paper dated the same date, predicting 50 different winners of the Australian Open. Yeah, like one scenario is a real prediction, and the other one is just, you have a bunch of guesses, and obviously one of them is going to be right. So it's relevant to the reader to know what else you tried to know how many times you had to try in order to get this beautiful result. So this paper is saying that some psychologists like are doing the equivalent of writing down 50 predictions on 50 pieces of paper and just not publishing when they predicted wrongly. But then critically, this paper then goes on for a while, just listing a bunch of ways, like a bunch of like acceptable methodological ways that researchers can like implement and get flawed results. But to be clear here, it's not usually as blatant as this Australian Open Example. What it looks like in the real scientist lab is not nearly that malicious or intentional or extreme, and you have a rationale for it. I think Samine is right. A lot of this was done in good faith. Some of these methods that can be used are money saving, you know, some of the methods are used to like just find something to publish and publishing is like the currency of an academic career. There's a lot of pressure to do it. Samine, for example, remembers going through her dissertation results with a mentor. And we picked out specific results and were like, these results together would make a good story. So she built that story and put it in a paper and then published it in like a really prestigious journal right before she got tenure. And then when this stuff hit, I was like, this is exactly what I was doing, pulling out a pattern from noisy data, highlighting just that pattern and ignoring everything that didn't fit and then telling a story around it. And it's not just Samine coming to this like really quite unsettling revelation. A lot of people are starting to say, wait, something's not right here. We need to check up on our work. Okay. How do you do that? How do you check like years of past research? It's called replication. All replication is saying, oh, you think you made a discovery? Let's make sure that it's a real discovery. And to do that, let's repeat it as closely as possible to what the original researcher did. Let's make sure that we can consistently get that result, maybe not a hundred percent of the time, but at least most of the time. If something replicates, it doesn't like tell you anything about how something works. It just tells you something works that like there's a there there. It's really foundational. These are the bricks that science has built on. You know, the 2011 events made us think maybe it's all a house of cards. So the next step was to take each brick in our house, each of the findings that we think we're building on and make sure they're real findings. Psychologists basically crack open site 101 textbooks. And they also think about like a lot of like the most popular findings in their field. And they just start asking, are these experiments replicable? Things like the 10,000 hour rule. It looks like we need to spend 10,000 hours practicing before we get good. Power posing. Stances researchers said would make people feel more powerful. The marshmallow test. An elegant, simple way to test a person's ability to delay gratification. Psychologists start testing these famous studies in a lot of cases with a lot bigger sample sizes and much more rigorous methodology. And a lot of them just don't really hold up or their results just seem much less impressive. Some studies have started to cast doubt on this whole ego depletion thing. One of three scientists who conducted the research said quote, I do not have any faith in the embodied effects of power poses. So you found that of these hundred studies in most of them, they did not confirm the original finding. Right. That sounds like a crisis. Yes. This whole episode has been called the replication crisis. So all of this is coming out. The studies are failing to replicate. How does the whole institution of psychology respond? Not well. Okay. Members of some of the old garden psychology start to get really heated. They're feeling attacked. This is their work that's coming under scrutiny, the work of their colleagues. I found one symposium like a series of talks from 2016 that has some of this tense flavor in it. Our symposium will touch on no less weighty topics than the foundations, methods and implications of science at this symposium. this famous psychologist gets up. His name is Dan Gilbert and he says the symposium here it's all been friendly but this friendliness is an illusion. This war he says it's leading to some of his colleagues like getting really stressed out and retiring early. Middle-aged people are screening each other at conferences and hiring attorneys and he directs some of the blame to just young people and their attitudes. Today it seems to me young psychologists are remarkably unkind self-righteous, intolerant, jealous, heading and snarky. This summarizes a lot of the pushback I've heard to this replication crisis that it's really about tone that people are just being really rude about it in public. But it can't just be about tone, right? It seems kind of weird to say psychology is falling apart but you all could be nicer about it. I guess I'd say some people think it's overblown but actually in the year since this symposium more and more researchers are agreeing. This is a real problem. Samine says as they do this big roundup as they check the bricks of their foundation that their science is built on, they find that maybe 50% of psychology papers replicate. That's not a house. You can't build a house. You can't extend on a foundation where less than half of it is real. This isn't an authoritative number. Very genuinely. We just don't know how many studies meet this basic test. There's just been a ton published. That'd be a lot of to check up on. But this 50% number is realistic enough to be scary. So does that mean that 50% of psychology is wrong? This is a really tough question. For one, yes, there's good reason to assume that a lot of psychology might be wrong. But this is the unexplainable. We really don't know how much. If we don't want to have all these questions remaining about the validity of psychological science, so much work needs to be done just to rebuild this foundation. How do you do that? How do you rebuild an entire institution? I'll introduce you to someone who's trying after the spring. This is Advertiser Content from IBM. My laptop works fine at room temperature and most things on Earth do. If quantum computers can only function and behave quantumly at insanely cold temperatures, how will they ever become useful to most people? My name is Olivia Lanes. I am the global lead for content and education at IBM Quantum. I want to emphasize that actually cooling down a quantum computer is not really as hard as I think people think it is. It really is just the push of a button that can take your entire infrastructure down to 10 mili-calvin. That is not the challenge scientifically speaking. I also want to emphasize that in the future we don't envision everyone having their own quantum laptop. Quantum computers are going to live in a data center. Dana Seniors already have their own cooling infrastructure. It's not going to be something that you have to learn how to install and work on a personal level but that doesn't mean that the discoveries made with the technology won't affect every person. Quantum computing has the potential to touch almost every industry that you can probably think of. If you would like to learn more you can do so at IBM.com/quantum. Support for the show comes from Google Gemini. You know that moment when curiosity strikes? You're exploring a new city with the family and a sculpture catches your daughter's eye. What is that? Who made it? With Gemini you can snap a picture and get a fully interactive breakdown of its history, architecture, and even hidden secrets. Gemini is built for moments like that. Whether you're learning something new, solving a problem, or looking for your next favorite thing. Google Gemini is here to help. Download Gemini, Google's AI app, or visit Gemini.google.com to learn more. We're back Brian in the first half of the show. You were talking about how maybe 50% of psychological research has been called into question over the last decade. It's all part of something called the replication crisis. So where does psychology go from here? Yeah, there are a few directions. And one really interesting one is from this guy, Chris Chartier. I'm an associate professor of psychology at Ashland University, and I'm a social psychologist by training. Chris's story also starts in 2011. He read the Ben paper, and he read the methods paper, and like some mean it made him wonder about his own work. And I even asked him like when you've looked, did you think any of it was bogus? Yes, bogus, dude. Maybe that's the wrong word, but like, did you like, did you, a fraud? Yeah, yeah. Or was reported incompletely or selectively against in spite of my best intentions. I would say yes. Chris did see these problems in his own work, even though it hurt a little to look. And he also saw these problems in the field of psychology. And he started attending these conferences where a lot of people were talking about replication. And then in 2017, he really, he had this epiphany. Do you remember the great American solar eclipse? The first total solar eclipse of the century, all across America, an unforgettable moment. Sure, I remember well, had my paper, paper glasses. E-clips glasses are very good, except for driving. So a few days after the eclipse, Chris wasn't wearing his eclipse glasses, but he was driving, he was, he was driving to go mountain biking. And he was thinking about the eclipse and just feeling jealous. Yeah, physics envy. That's the like Freudian phrase that you could toss in there, I suppose. He started to think about like what led to this moment. How much physicists and astronomers had to know about the solar system and how right they had to be and how precise they had to be to predict eclipse? Like they allowed us to have this grand moment together. And then he just thought about his own field, psychology. They do not have this precision. Then I took this three-hour mountain bike ride out in the woods, spotty cell service. It's a great place, time setting for me to think. And the contours of this idea that emerged as the psychological science accelerator started to take shape. Sorry, the what? Yeah, the psychological science accelerator. This is Chris's idea to make psychology precise, eclipse level precise. On the drive home from that mountain bike ride, I recorded an audio memo to myself. Oh, it would be a real trip to listen back to my audio memo. Please tell me you got this voice memo right? Oh, I have it. Um, blog post building a SIRN for psychology. Make the pitch that Chris wants to imitate SIRN, which runs this huge particle collider in Europe. And Chris is starting to get a little technical here, a distributed laboratory network. But the gist is that he wants to fix psychology's crisis by bringing together lots and lots of researchers from all around the world and form this like one big testing squad. So he he gets home from his bike trip and he he just starts on this. So I'm still in my bike shorts because I was like too frantic to shower. He writes his blog post like this kind of mini manifesto of like this dream he has. And he puts up a sign of sheet and you know, as as a young researcher does, he puts it on Twitter. Really? It was a classic academic viral experience. I always put the academic qualifier on there. Like, I think the tweet was retweeted like 30 times. And I was like, Oh my God, Kim, I'm famous. We made it. And so like the yada yada yada here is that now there are more than a thousand researchers in this big network. And they come from more than 70 countries all around the world. It switched from like, this is a cool idea. Maybe 10 people will do it with me very quickly to like, we really are going to do this thing. Okay, that seems like a like a great idea. You know, lots of people all over the place working together. How does this actually solve the problems at the core of psychology though? Yes, there are a few things here. One is that they're going to do huge experiments bring together lots of people have huge sample sizes internationally, you know, test things around the world. But the kind of main, like the key ingredient here is called pre-registration. Pre-registration? Yeah, remember that this whole big problem starts with researchers just tweaking things as they go along. So pre-registration cancels that. It's basically before a researcher does anything. They write something down. It's like a recipe for a study. For our souffle, we will use three tablespoons of butter. They list all their ingredients. Three egg yolks. They list all their steps. We must butter our souffle dish. They list like how they'll analyze the results. We're looking for a crispy brown top and for four inches of rise. And you know they they submit this. They put this out in public. So other people can like look in you know keep them accountable. It prevents researchers from tweaking things along the way. They can't do a oops. I forgot to publish these 49 failed predictions. You know that just can't happen. This is my plan and I'm sticking to it. Could you just still tweak the recipe and just kind of keep it quiet? I ask Chris the same question. Yes, you could still do that, but it clearly makes that fraudulent. Okay, fair, fair. So I guess you have all these labs all over the world making these promises, you know, to stick to these recipes that they lay out ahead of time for important studies. That's pre-registration. That makes sense in theory. I get it. What does it look like when you actually try to make it happen? Well, you know what happens when dreams meet the real world. Oh dear God. My kitchen is such a mess. Chris asked people to email him and he asked what psychological studies should we look at? You know, should we check up on? So he wanted people to submit their own research and ask people to prove them wrong? No, it was not bad. Yeah, people were submitting other researchers studies and saying, let's check this. Okay, that's awkward. It is awkward. Yeah, so they get these potential papers, these these potential experiments to check. And the accelerator group had to decide which one should we test? And they did pick one. The first person was a researcher at Princeton University, Alex Todorov. He's kind of a big deal. He has a really influential idea about how we make snap judgments from people's faces. It really helps explain a piece of where stereotypes come from. And so Chris reaches out and tells Alex, hey, we want to test your idea. But here's the beauty in the accelerator or maybe the beauty in Alex or both. His reaction was positive. Well, I remember, of course, I remember it very clearly. That's Alex Todorov. Yeah, I called him up. I really just wanted to know how he felt when Chris reached out to him. Everybody has been talking about replication crisis. And often when people go to replicate something, it's because they don't believe in it. And you know, and that can put you somewhat uncomfortable, feeling uncomfortable. But Alex is pretty confident. And he was great in those initial interactions, just enthusiastic, supportive, I think excited. This was his chance to test this idea all around the world. And I've always been interested, but it's impossible for a single lap to do this at this largest scope. So Alex gives advice. And then Chris and his team do the super critical step pre-registration. They contact a journal and they send in their whole recipe for the study. Our methodology. Three eggs are data collection plan. Three tablespoons of butter are analytical strategy. Six cups of grated cheese. And then it starts. They start testing this idea around the world and data starts coming in and they start analyzing it. Remember, if the top isn't brown and crispy, then it is not a souffle. And then there's a snack. Yeah, so remember here, Chris and his team have their recipe down. They've committed to it. But then Alex Todorov, he starts to do some reading. And he starts to have some doubts about one of the final steps in the recipe. Wait a second. If the top gets overly brown, then the souffle might be burned. Yes, them. Can we tweak this? Isn't that like the whole point of this whole pre-registration thing? No tweaking the recipe? Yes, absolutely. That is the whole point. But Alex makes an interesting argument here. He says you don't always know everything you're going to need when you start a project. Imagine you have brain surgeon and you pre-register all of the steps of your brain surgery. And then you started poking in the brain of your patient son and said, oops, if I don't do this, I'm sure you're going to kill him or her. Would you change the procedures? Okay, what is Chris saying that? Oh, he didn't he didn't agree. He and his team, they didn't think there was a problem with this recipe. And they didn't want to let the person who had the most to lose here coming in federal with it. So they refused. I would say that the initial enthusiasm ran out during this process. It sounded really tense. There was a lot of back and forth. Eventually this paper published. The replication results came out. And honestly, they were kind of messy and split. It sounds almost like Chris had this big, beautiful idea. You know, everyone will just do pre-registration. Promise not to tweet their recipe and everything would just be fine. But I guess in the end, if you actually try to make a certain for psychology in the real world, it's just it's just a lot harder. Could there be a better demonstration that it's needed? A pattern of human response where initial enthusiasm gives way to debate and argument only once the data are in hand is kind of a really elegant demonstration of some of the problems. And I guess Alex is an example of one of the problems here. I don't want to demonize Alex. You know, you there are rude scientists out there. I have talked to them. He is not one of them. He was very generous with this time and he was very thoughtful. But this is just really hard. It's really hard to invite people to invite others to look for flaws in your own work. People always talk, I mean, the stereotype of the scientists, especially from an outside perspective is like, okay, we object if we have personal life and the science is completely different. And that's just not true. I mean, have a few life is your work. And then you built in your reputation. Science is done by people and people are going to have emotions about their work. It sounds like the method to fix psychology is really, really complicated and difficult. And it's going to take a long time and tons of people are going to burn out. And in the meantime, like we still have psychology, right? I mean, can we trust psychology right now? I actually think psychology is leading the way on a lot of this. These problems we've been talking about with like cherry picking data and like the need to publish at all costs. This exists in other fields of science. What like, like, a ton of biological research and chemical research isn't true either? Yeah, go Google replication crisis and biomedicine. There's one there. It's just less well documented. Psychology is really paving a path forward here. And right now, we don't have to throw out everything in psychology. We have learned things. There are, there are studies that replicate. But good science is a gift we give to the future. So we have to get it right. If you can just marginally improve the way we collect and analyze our data and draw conclusions from them, there are untold future human beings that can benefit from that tiny advance. You don't have to answer all the questions today. It's good enough to make incremental the lasting progress on how we ask those questions. And if psychology can find a way to make those incremental changes, maybe other fields can replicate their success. Brian Resnick is a senior science reporter for Unexplainable and Vox.com. This episode was produced by Bird Pinkerton. We had editing from Meredith Hodnott and me, now I'm Hasenfeld. Liliana Mitchell and I did the fact-checking. Hannah Sprown did the mixing and sound design. I wrote the music and Liz Kelly Nelson is the editorial director of Vox Podcasts. For more about the accelerator and their plans for the future, check out our newsletter. You can sign up at Vox.com/ unexplainable and as always you can find a link to subscribe in our show notes. And while you're over at Vox.com/ unexplainable, we've got something new for you over there. In case you want to share any of our past episodes with kids or students, episodes that have curse words in them, check out our site for a link to clean versions of past episodes. And also email us. We love reading emails. We're at unexplainable at Vox.com. Unexplainable is part of the Vox Media Podcast Network. And we'll be back in your feed next Wednesday. And Edward Jones, we believe Rich, is about taking care of what gives your life meaning. That's why your financial advisor personalizes your plan to help you preserve your progress and create something that lasts. Let's find your Rich, Edward Jones, member SIPC. For a limited time, you can get a big Mac meal for just $8. That's a burger, fries, and a drink. They don't call it an extra value meal for nothing. Get a big Mac meal only at McDonald's. Pricing participation may vary, promotion pricing may be lower than meal pricing.