Warren Hatch 'Superforecasting - how to see the future first'

[Music] Hi and welcome to my podcast series The New Abnormal, which looks into now next why. I'm Sean P. Lodishanasi, a futurist and strategic consultant who works for governmental, academic, corporate, agency and NGO clients. I've spent 20 years working on catalytic international projects across a wider array of sectors. Along the way, I've been lucky enough to emit some amazing people from psychologists to activists to creatives. Some of them are guests in this series, along with an array of other fascinating individuals. I'm also the author of the Post Truth Business, which focused on trust, while my second book, Influences and Revolutionaries, focused on innovation. As a public speaker, I've given speeches at events from San Francisco to Beirut, Melbourne to Stockholm and Tokyo to Paris. So, I hope you enjoy this podcast and I'll now introduce my guest. So today, I'm really looking forward to talking with an incredibly interesting individual who I've been wanting to interview for a long time. Warren Hatch, who joined Good Judgment as a volunteer forecaster in the research project sponsored by the US government, became a super forecaster, and is now CEO of the commercial successor Good Judgment Incorporated, a world leader in applying innovative probabilistic solutions to real world decisions to forecast the future. He's assisted governments and private companies around the world to improve their foresight and quantify uncertainty. Warren's prior career was on Wall Street, where he started at Morgan Stanley before co-founding a boutique investment firm near his PhD from Oxford University, and there's a chartered financial analyst, Charter Holder, all absolutely fantastic stuff, sounds deeply impressive. So, hi, Warren, and how are you? Hi, Sean. I am fine other than being called here in New York City. How are you today? I think slightly warmly the new, but in a rather grey brightened, but there we go. I mean, Warren, there's so much to talk about, and I'm really, really interested to hear your perspectives on all of these issues. So gone, then has to be asked, first of all, so there you were, perhaps, at Oxford back in the day, how did you actually get from Oxford to where you went out? Well, at Oxford, I was studying Soviet economic policy making, and while I was doing that, I went from being a political scientist to an historian, as my country disappeared. So I needed a retool, and people need to do that often, and I knew someone, who knew someone who needed someone to do research and go striding on Wall Street in Morgan Stanley, and that was the chief investment officer there. And he, in addition to being a manager of portfolio managers, he was put out a quarterly piece to retail investors, and his thinking was, who better to write on the markets for people who don't have a deep familiarity with the markets than someone who does not have deep familiarity with markets? So my lack of market knowledge got me the job. And while I was there, though, I got super interested. It was a great opportunity. It was kind of a college atmosphere where I could just walk the halls and talk to different people, doing different strategies, and eventually got my qualifications for a CFA, and became a co-puff portfolio manager with Joe McLean and the CIO, and went from there. Fantastic stuff. And in terms of what you actually do now, perhaps I'll just put it in the context of other futures thinkers. So well aware in terms of good judgment, home of super forecasting, helping clients quantify subjective risks for better decisions. Let me just put a classic futures quote out there. So one of the many William Gibson, who I interviewed back in the day, who said, "We're following. We have no genuine idea of what the future may hold because our present is too volatile. We have any risk management and the spinning of the given moment's scenario pattern recognition. But then in walk you and the super forecasters. So over to you in terms of what you do now, you do it." Well, I think taken as you just read it, absolutely, we do not have a crystal ball. We do not know with certainty what the future will bring. But we can quantify our subjective judgments about what the future will bring. And that is the key thing. So it's a lot like with the way weather forecasters used to be delivered. Now decades ago, the weather forecast, well, there's a fair chance of rain next weekend. No one would accept that now. They want to know there's an 80% chance of rain next weekend. So you can decide whether to proceed with the picnic or not. And this is basically the same thing. It's like the weather channel for geopolitics and anything else where there's a subjective future. And far better to quantify it, it's more impossible, while recognizing their limitations to how much of that we can do. But it's better that than deferring to someone who puts their finger in the air and says there's a fair chance of war in the Middle East. Yeah, but I love that. The analogy of the weather and the picnic. And I remember reading something you put out a while ago and talking about the difference in importance, perhaps, when it comes to weather, whether you're planning a picnic or a wedding. Exactly. Yeah. So 80% chance of rain or say 20% chance of rain next weekend for a picnic. Maybe you just go for it. But if you're going to be having an outside wedding reception, yeah, you might want to bring out the tents. Yeah. And so these super four consters then, so Gondyne, Warren, who are they? And what is it about them and you just different to the rest of us? Well, you can't look at somebody and say, "Ah, there's a really good forecaster." It's not like somebody in sport where you know, you can say, "Yeah, this person looks pretty athletic." And the only way to find out how good somebody is at quantifying uncertain events is seeing their track record. And by that, it's like, if I say there's an 80% chance of something occurring, it should occur 80% of the time and not occur 20% of the time. So four out of five times, you'll look like a genius, one out of five, you'll look like an idiot. And that's being well calibrated. Then you can have confidence in that 80% number or whatever the percentage actually is. And they're everywhere. Every organization has people in their ranks who are going to be very good at this. And the only way to find them is to see their track record, give them an opportunity to provide those forecasts and see how well they do. Now, there are some characteristics that can help you kind of pre-screen. Good forecasters tend to be good at pattern recognition, which is exactly what Gibson said. They tend to question their beliefs about the world. So rather than protect them against new information, they'll re-evaluate them when there's new information. And they also, for things that really matter, slow down their thinking process. And Daniel Coniman calls it system too. Like system one thinking is what you can be driving down the road, fiddling with the radio. You can do all those kinds of things on autopilot. But when you want to think about something that's really important, rather than go with the first thing that comes to mind, you want to challenge your thinking. Have it even have an internal dialogue with yourself. And that's going to give you a better number. Good forecasters do that. And doesn't this sort of link perhaps on the first ethical level to thinking of those like Karl Popper that was very much into challenging thinking, challenging the accepted view? Absolutely. Yeah. And the way we pose questions about the future should be falsifiable. Absolutely. That's foundational. Yeah. And again, I can say I'm just quoting you feel like you do pack at you. But I mean, other stuff I've seen you put out is so much when it's so fascinating. I saw some research done by UC Irvine talking about super forecasters. And amazing statistic, yes. And anticipated events 400 days in advance, accurately, as opposed to regular forecasters who did it or could only do it 150 days ahead. I mean, in terms of the focus of that and the implications, that's fairly massive to put it. Absolutely. And that was done by an independent academic who evaluated all the data on forecasters that were collected over the course of a year. And that was the key conclusion is the people who are a skilled at this and be apply process to this can get to the best number possible faster than other people. That is the advantage. Time is money and having that early insight matters for everything from sales, marketing to military and politics. And in terms of you talking about sort of the keys to this, I mean, is there a set process, there's sort of a checklist that one goes through? Everyone should have a checklist. And they'll find some things are more useful than other things based on what they are doing in the topic at hand. But there's some things that are worth having in all checklists. And the main one to start with is when first presented with the topic, say, you know, who's going to win the next election in the UK, people will tend to get caught up in the specifics. And who are the personalities? Where are things now? This is again, what economy calls an inside view. But the thing is that starting with where things are now doesn't give you the best possible number because you may need to change your view. And we become anchored. And so wherever we start, we want to start in the best possible position. And the best possible position is starting with how do things like this usually go? What does history tell us? This is the outside view or base rate. These mean the same things. How do things like this go usually go? And then how is this time different? Most people tend to start with this time is different. Whether it's a stock or a sales plan or a project, whatever it is, it's far better to start with. So that's step one is how do things like this usually go? Step two that's very good to do is to compare your reasoning with others. Do it on a team. We all have partial information by definition. We can't know everything. But you might know a bit of the puzzle. I might know a bit of the puzzle. Others might know. And then we can put it all together. And it also helps us evaluate how strong the information is because I might think, oh, this is something really important. And you'll point out, well, you know, actually it's maybe not that important. You know, you know what, you're right. So that leads to the third thing. And that is to change make updates when that circumstances warrant it. You change your mind, make an update. And then the final thing is eventually you're going to get an answer. We'll look back and say this happened or it did not. That's an opportunity to get your score and evaluate how you're reasoning aligned with reality and learn and get better. And by doing all of that, you can get the feedback to come up with better numbers faster. If one of the risks is if you don't keep score, you don't keep notes of what you're thinking was along the way. When something happens, people tend to go, oh, well, you know, I kind of knew it all along. Hindsight bias, this is called. But if you keep a track record of what you're thinking was, what your forecast were, you can hold yourself accountable and get better. How interesting. And so you say, so if you like the the really experienced super forecasters amongst you and your team inside just say internally and externally, they always follow that sort of that that teamwork approach so they're not doing it on a solo basis. They are very much, you know, doing it literally on a team basis. It is very much a team we know from the data in our experience and others too is that a team of skilled forecasters will do better over time than any single individual. That's that's the way it works. And we all know that. And so we seek out people to challenge our thinking to make ourselves better. Now sometimes you need to do something all on your own. You can't really have others to help you think things through. And that's that's where you can go back to that internal dialogue because you can have a team. I can have a team of me and myself and challenge my thinking and go well. So this is where I think it's going to go but if reality goes a different way, what might I be missing? And you can just do that on your own. And in terms of challenging thinking or if you say sort of catalysts around thinking. So at the moment, any particular things that are particularly interesting you regarding today, tomorrow the future hasn't sort of horizon one two three is some way. But I can say that so in our subscription service, we have the topics nominated by users. And right now these days the top topics have to do with conflict. Existing conflict or threat of conflict. So how will Russia and Ukraine unfold? How will the Middle East unfold? What about China and Taiwan? And this again is where it can be useful to look to history to kind of have a starting point. And what we know from from history is that there are lots of hotspots around the world. But in any given year, just a few of them may flare up or none. Going from a hotspot to conflict doesn't happen all the time. But we also know now we start taking an inside view is that in recent years, political leaders have had more appetite for taking risk. And on the presumption that they will have the upper hand and get a quick victory. And sometimes that happens more often than not though rather than a quick victory becomes a prolonged extensive conflict. And that's I think what we saw with Russia Ukraine. You didn't certainly didn't think he thought it was going to be you know over in a weekend. That was not the case. So we also know that once conflicts begin they last a long time. And again in the case of Russia Ukraine, there have been these periodic bouts of optimism that the conflict would come to an end somehow. And the headlines flare up. But then they fade and things don't really change. So when we think about that particular conflict, in our view we're quite skeptical the BSCs fired anytime soon. Similarly with the Middle East. And then coming from the other side, China Taiwan, another hotspot. We remember that hotspots flaring into open conflict. Rare, we also know that the political leadership there seems to have more appetite for engaging in things. So the risk probably higher than they are historically. But our view is still as a group, the prospects for conflict in that part of the world are relatively low. And those are the things that people are really watching right now. The other big areas has been US tariffs and how that's going to go. And then also election outcomes. There's some pretty big elections coming up. So those are the big things in the subscription service. Yeah, yeah. As interesting as you see, but only today I think we've seen the Supreme Court to just struck down Trump's current viewpoint, I believe, or where things are tariff wise, but we'll be fascinating to see what happens next. You've missed. Oh yeah, yeah, it's history in real time. That one. Meanwhile, in terms of other big topics, and I know you've been looking at this as you would in a very, very big way, but going in, human versus AI. That is a great, great, great topic. And our favorite Boolean, certainly mine, and I think most of my colleagues is Amp. So it's not an either or it's both Amp. So the tension between AI and humans in many ways, I think is an artificial one. And again, if we look to history, the having humans work with machines, that's what's been going on for millennia one way or another. And certainly in the information age, we've just seen an amazing acceleration of that. And the word computers is one of my favorite example. It used to be a human sitting in a back room on an adding machine. But then the machines got better and took over the heavy lifting. So that freed us up to do what we're good at, which is judgment. And so for AI, in our view, is it's not a new thing. It's certainly a very rapid acceleration, phenomenal acceleration of the development of machines to do a lot of the heavy lifting for us. However, in our judgment, the AI is not going to replace humans in all things. Now where there's a lot of data and it's pretty stable, then great. Let AI do it. It can do a really good job so we don't have to. It's still got some problems, but certainly an enormous amount of promise. But when it comes to converting that, the subjective hunches about the future, where the data is sparse and the environment is in flux, what we see is you still need to have humans. So you can have both. AI can help you in that search for history and comparison classes, the base rates, and can even help you synthesize it. But AI is terrific at synthesizing existing ideas, creating new ideas still seems to be the domain of humans. But we can do it better without a cyst. We also know that when we're having so for people who prefer the ore, there's only one competition right now that includes super forecasters and AI, and at least to date, the best of the AI is still 40% behind the best humans on real world issues. Which is an extraordinary statistic, I think. So that's what I think. We've already, really surprised many of those who would assume that AI is in fact leaps ahead, but absolutely not the case you're suggesting. In certain things, absolutely. So if you're trying to figure out what the monthly inflation rate in June will be, I would happily defer to an AI. If you want to know which party is going to win the next election, I'd want to have humans in that loop. Just to be clear on the say that I think as you put the four keys to accurate forecasting, which is really the way that you've written about in the past. So just be clear about this in terms of the talent training teamwork, aggregate angle. Just talk us briefly through that. It's such an interesting, again, sort of a guide straight checklist. Yeah, so the researchers led by Phil Tetlock and Bar Mellers in the research project, they came very open-minded. The task that the US government had set was improve on the wisdom of the crowd. Because they wanted to improve the skill set of their analysts and invited five university-based teams to see what they could do. The other teams had tended to have a big idea that would really give them the edge. And the good judgment team said, "Well, you know, we really don't know what works. So let's just try a lot of things and see what you get when you put all those things together. Some things are good contributors, but some things aren't." And knowing what not to do is as useful as knowing what to do. So you can focus your energies accordingly. And what they found was, yeah, so there's four basic factors. The first is some people are just better. We were talking about that earlier. Out of the starting gates, they have those characteristics of pattern recognition and active open-minded thinking that set them apart. And by identifying the right people for your team, that can be a big contributor before you do anything. The next thing, though, is that anyone can get better if they really want to at this. So having proper training, like some of the steps we were talking about earlier, start with base rates and make up those sorts of things and be aware of what kinds of cognitive biases that interfere with the judgment of the sort. And what you can do to mitigate those effects is a big contributor. So have some training for yourself and your team is a big addition. We also know that having teams will accelerate the process and give a better number. And having a well-structured team gives you that big advantage. And then the final thing has nothing to do with the forecasters. You can have algorithms to squeeze out more signal and dampen more of the noise to get a better number. So for instance, a lot of people tend to be over-confident to assign higher probabilities to things, especially in their areas of expertise, then reality actually shows. But they do it in a regular, predictable way. So you can have an algorithm to say, okay, when this group gives a forecast at 90%, it's really going to show up 80%. And you can have an algorithm to help you get a better number. And also in terms of futures literacy, helping your clients be futures ready, etc. No, naturally, one goes back to the foundation of your organisation and the mighty, much-respected Philip Tetloff. Are there any other particular futures thinkers or futures techniques that you would naturally or often refer to? Or do you tend to stick very much to Tetloff and your sort of a cool process? Good forecasters are going to be intellectual magpies. They're going to read voraciously and take ideas that help them. Amanda, ones that don't. So I think most good forecasters will have a pile of books by their bed that they're still going through. And that's certainly in my case. I think the main thing is, and again, it's complementary, is it futures frameworks and scenario frameworks and there are a lot of related terms to think about the future. Most of the time, they focus, they first they start on all the possible worlds. What's possible? And then they'll narrow the focus to the plausible worlds. And they'll put together a story about how the future will unfold. And that can be a great way to be prepared for different kinds of possible futures. So when they start showing up, you got kind of a playbook to follow. But what we do is take it to the next step. What are the more probable worlds? Which ones are more likely to unfold relative to the others? And then we can focus our scarce resources on what's more likely rather than what's more plausible. And this is a really important and sometimes subtle point is stories, which is what a lot of scenario analysis will do, put together a story. Stories make our brains go, ah, okay, that makes sense. That's the way things are going to go. But stories tend to low people and get them to think that that scenario is more likely than actually proves to be the case. So what we do when we go from that plausible to probable is we'll strip away the story. What's really going on here? What are the real drivers here for this potential future? And then let's think about the probabilities for those drivers stripped away of the story. And now we've got a better sense and more unbiased to view. We have mitigated the effect of the narrative fallacy. That's one of those biases. And then now we've got a better view about the future. And we can package it back together into a story because to get people to do things to motivate them, you do need a story that's the way it works. But now you've got a story that's got much stronger underpinnings. So interesting. And how to say, you mentioned that you will be about sort of a part of books by your bed. So I'm actually very inspiring individual words. So who inspires you? You are great reader, listener, watcher, etc. And if so, what and who? Yeah, well, um, Deft definitely got boxed by the by the bedside. And one that's on the top of the pile is this new one, 1929 by Andrew Ross Sorkin. Yeah. I have yet to crack it open that I look forward to doing so soon at a long plane ride coming up next week. And the other one that I just put back, because I took an old friend off the shelf and put it on the pile, is weathering heights, because they just released a new movie and I go, well, maybe I want to see it, but that's one of my favorite books. Yeah. And so I thought I'm just going to take a fresh read. It's just really good. And then I've certainly listened to some podcast. My favorite is in our time. And I'll miss Melvin Bragg, you know, that he's stepped aside, but that's just been one of my favorites. Yeah, absolutely. Genuine genius. I think we'd all agree on that. And then it has to be said. So in terms of, you say, there's a future going in in terms of you then, Warren, what's what's on horizon in the sort of short to meet in terms for you and the company? Well, we have a mission to bring probabilistic thinking to improve decision making wherever possible. All right. So and one way to do that is with AI. So we like everyone else. We are incorporating AI into the process. We've evaluated a lot. There have been a lot of startups that claim to be super accurate with the four. We haven't been as impressed as some people have with performance. So we're building our own. So that's one thing we're doing. And we're going to make as much as possible, at least the foundation parts open source. So anyone can use them and add features that are going to be useful to them. And we'll be doing the same. So that's one thing on the technology development side. And then we're also engaging with with media to encourage the use of hard probabilities rather than vague language in the coverage of these important events and continue to do that. And and then provide training. So we have the best forecasters. So people can outsource their forecast to us. But we can also provide training for them to develop their own internal talent. And and that's been quite exciting to do to see a lot more traction lately with from big companies to small NGOs that really want to bring more rigor to their internal decision making. So that's that's what we're going to be doing in the very near term. How interesting. And as I mentioned there the whole issue around AI, just asking if you could just a couple of case histories of that giving anything away that's sort of confidential. But I noted a couple of things that you highlight. So AI governance and then also US government health funding to very, very different issues. One link to safeguards and the other perhaps in the context of obviously American government dismantling USA. So we're going in first of all. So AI governance and safeguards. Yeah. So we've been involved in a lot of different aspects of AI. One aspect is what kind of a threat is it going to be in the future? And is are we looking at SkyNet? And there are a lot of people who argue very persuasively they tell a great story that the risk of AI is already here and it's too late. And all we can do is humans is try and bend the direction of artificial intelligence for it to be as benign as possible to humanity rather than malign. And we did a project a little while ago that I found really fascinating. And that is we had a group of AI experts who were quite convinced of this and perhaps some day they might be right. And we had them we put them on a team with super forecasters and the part of the goal was to make forecasts on what those risks are. And then also to see if that exchange of views could get anybody to change their mind. And because a good part of the forecasting process is to say, well, if this happened I would change my mind. It was quite interesting that the AI experts didn't have anything that they said would change their mind. They hadn't read properly I guess. >> Oh, there we are. >> Yeah, but the super forecasters didn't change them. But the AI experts, obviously very high probabilities, super forecasters, quite low probabilities on that kind of risk. So when we think about AI governance, the problem is there that may not be as menacing as some of the experts think. But it's not to say that there shouldn't be good controls. There should be. But how we think about them I think might be a bit different. And then the health spending. And this is something that's not just the US. It's a global thing. And I had a wonderful opportunity to speak at the UN in a small, it's not not the general assembly. But Oja, to evaluate that because they're quite concerned about the global reduction in spending on healthcare and humanitarian aid. And when might that change if ever. So on our site we do have forecast for when the US in particular, or what the spending trends by the US in particular will be. And it's not going to change any time soon. Wow, how interesting. I'm aware by the way of times, I'm aware you're a very busy individual warrant. So it has to be said in terms of perhaps all of the summarising. So how yeah, some sort of key points that you'd like to leave regarding understanding today, anticipating the future with you. Well, thanks. Yeah, well, I think a major takeaway is rather than, for things that matter, rather than use language to describe how the future will be, or only language, attach a number, convert it into a number, and change that number when your mind changes. And in that way, we're all speaking a common language. If people are saying, well, there could be something or there's a distinct possibility, it's really hard to put that all together and keep track of it. If we say 72%, even though that's a false precision, 72%, come on. But at least now we've got something we can all understand. And the other really useful thing to do is when politicians and pundits in particular express their views about the future with that kind of language, with a story, hold their feet to the fire. What are you really saying? As I say, there's a distinct possibility of something occurring. They're going to be right no matter what. If it happens, see, I said, if it doesn't happen, I said, well, only a distinct possibility. So hold their feet to the fire. Hold themselves. Hold them accountable. And hold yourself accountable too. And you'll be amazed that how much better you can be at converting subjective beliefs into something quantifiable and much more useful to lead to better decisions. And ultimately, that's what this is all about. We want better decisions. And all decisions, in a sense, are forecasts. We're going to take a course of action that we think will most likely lead to our desired outcome. And by applying process and using numbers, we're going to have far better odds of getting our desired outcome. Nicely put, it has to be said, but fantastic. And so just so everyone is crystal clear, where they can track you down online to read your words of wisdom and they don't get in touch with the company. We're absolutely. Yep, there's we have our main site, goodgegment.com. Everything can be found there. A lot of resources. And then we also have our public forecasting site. Goodgegment open. And that's where we've got a lot of questions there. A lot of challenges, groups of questions, including one by the economist that's been very popular, where people can go and try their hand. Because it's one thing to read a book, another thing to hear somebody talking about it. And I could talk about it till the end of time. The only way to get better, like playing a violin, is to do it, apply it. So good judgment open. And we're also on LinkedIn. Fantastic. Well, I could have had to listen to you for a lot longer, but time is against us. So really just will bring me to the end of the conversation or bring us to the end of the conversation. Just leaves me to say to where I'll hatch the super forecaster, renowned speaker and CEO, goodgegment. World leaders and applying innovative probabilistic solutions to real world decisions in order to forecast the future. Thank you very much indeed. Thank you, Sean. Been delightful. Thanks for listening to that episode of the new web normal podcast. Just so you know, the issues discussed in these podcasts also linked to my speeches. Check out Seanpdacet.com for more details. And for information on my ongoing research, check out brandpositive.org. Please also listen to previous conversations across the series with many other fascinating guests. But till next time, goodbye.