PSPRS' Mark Steed: Unlocking More Powerful Investment Predictions / Decisions

(upbeat music) And we're back, more innovation without the threat of getting fired. I am Ashby Monk. I run a research center at Stanford University among many things. My co-host is Daniel Adamson. The adventure capitalist he likes to remind me, is that still true? Are you adventuring? - You know, that's what I tell my kids. It's still true. I am having so much fun. I'm adventuring through capitalism like never before. - Well, let me tell you something, Daniel. I go to venture capitalist websites. - Yes. - And I am blown away at their ability to predict the future. - You look through many of the top venture capitalist websites and you just see winners across the whole website. - I hate to bring it down here, but the thing that's common to all quote unquote venture capitalists, man, can they tell a good story about the future? - Yeah. - Whether that story comes to pass or not, you know, I think it would be valuable if we had a guest at some point. - Yeah. - We can work on that. - Who has some thoughts about, you know, how to predict things more accurately than a lot of these so-called quote unquote venture capitalists out there. - But before we get to that, Daniel, I wanna just say that the classic trick in finance is to make many predictions and then delete the ones you got wrong and focus on the ones you got right, make more predictions, then delete the ones you got wrong, focus on the ones you get right. And I have to tell you, I think that is actually what most venture capitalists do with their website. - I agree. I think when you look at a website of a venture capital firm, you won't see their failures on there. - Sadly, I think that's true of a lot of disciplines. (laughing) I think we've learned that scientific research may not be immune to that problem as well. - True. - True. - And I think it may not be a problem limited to venture or adventure capitalists. - That's right. - Within finance, I think, you know, if you look at a private equity firm website, and I just don't wanna pick out one cohort specifically. I think you're talking about a global problem here, which is revisionist history. - I think I'm talking a little bit about the challenge of making good predictions. And especially when you're a professional investor, the importance of communicating confidence and reliability in an industry where we're probably pretty darn good at what we do if we get some small percentage of our bets right. And we know how to size those bets. - Yeah. - And, you know, in an industry we talk about our gut, we talk about our crystal ball, we talk about our spreadsheets, all of which are riddled with errors. And so ultimately, as investors, we're trying to make predictions about the future. Inventure capitalists are making bold bets, many of them, but it's very hard to do. - Yeah, I mean, I know we have a very esteemed guest to introduce and people don't wanna hear from us for that long, but-- - Yeah. - I mean, people talk in the investment industry all the time about, well, it's an art and a science. You know, maybe, let's double click on that for a second. One of the things about art is art evolves and the old art is just as good as the new art. You know, if avaldi, no worse than Beethoven, no worse than Stravinsky, we appreciate it each in its own time. Science, however, is supposed to make progress and get better. Get better specifically at predicting what's gonna happen in the world. And I think it's an interesting question to ask if investing is getting better. - Yeah, I don't know. Are we actually getting better at it? I mean, the Venetians, pretty good. Double entry accounting systems. They had the arbitrage game going. That was 700 years ago. - Oh, 7,000 when you go back to the, what is now a rack, you were finding accounting on tablets. Clay tablets. - Yeah, so I don't know that, I mean, we like to think of it as a science where we're getting better, but I don't know. Maybe our guests will have some perspectives on this. - Speaking of our guest, he is the chief investment officer of a ZeraZone as public safety personnel retirement, Mark Steed. And he's gonna walk us through how he could get the coolest degree of all time, which is a masters of science and predictive analytics. I didn't even know that existed. - Wow. - Mark, welcome to the Don't Get Fire podcast. We can't wait to talk to you about the future. - Hey, thanks guys. Yeah, and you said at the beginning, it's innovation without the threat of being fired. And I was like, oh, I didn't know that was part of it. I feel like there's a threat every day. - Yeah, exactly. (laughing) - Yeah, yeah. But yeah, my degree in fortune telling, doesn't help me predict whether I'm gonna get fired or not. And unfortunately, Daniel, I'm not a good storyteller. There's probably more money in that. - There's a lot of money in storytelling when it comes to venture. But in truth telling, that's where you actually make good investments. - Yeah, I believe that. - But isn't it interesting how is an industry? Literally our job is to try to make sense of the future. Like capital market assumptions, scenarios, stress tests, liability management, all these things are about taking today's cash and pointing it into the future. And you and I have had a few chats, Mark, which is why I'm so excited about this show about your approach, which I think is unique. And we're gonna get into it. But before we do that, can you just tell us about PSPRS, your role? I know you've been there since 2007 and you've literally held every asset class ed role. - Yeah, I started there as an intern in 2007 out of grad school and I was at my MBA and started managing the private equity portfolio in venture capital at the time when I joined 2007. They looked like a lot of other pensions, which were very stock and bond heavy. And then went through the financial crisis and all the pensions said, we don't like that. And they looked at the endowments and said, hey, why don't we invest in alternatives, right? That was the idea, the endowment model. And so everyone went, hold on to the endowment model and they said, hey, do you have any interest in sort of helping us build the alternative investment program as somebody coming out of business school, got options to go sitting in cubicle somewhere and grind spreadsheets for 60, 70 hours a week or do something meaningful, like build a program from scratch with real money. So I started doing private equity venture capital and that was just like writing the policies. It was just, you know, doing the first basing model. I mean, there was like nothing there, you know, trying to find consultants and then we've always had this culture of rotating employees around the asset classes. So because you're running with very smart and staffs, so it's really important to have more than one person that knows what's going on. And so I ended up moving into fixing a credit and then over real estate and hedge funds and equity and here I am, I always thought I would leave after five years like everybody, you know, kind of think I'm going to sit here for five years and go do something else. But then after a while, the whole program starts to kind of look like your child in a way. And because you've touched everything and I started to realize that it's hard to find jobs where you're adding value and your employer recognizes that you're adding value and they're rewarding you for that value. And so I don't take job hopping lightly and I've just kind of hung out and started as the CIO and really at the end of 2018, the end of 2019. - Kind of makes total portfolio approach like almost just a natural outcome because you built the total portfolio. - Yeah, it's funny, I heard that term, the total portfolio approach, total portfolio management. And I just thought, what are we doing? I mean, I thought that's what we were all doing, right? Like looking cross asset class, the nice thing about having like rotations and this is just also a natural consequence of decision making that people don't feel like they own things and they're perfectly capable of saying and empowered to say, I don't think we should be investing in this no one's territorial. It's not like they have an asset class to protect or they're out of the job. So you can really pit like ideas and like risks against each other. And I think that by itself just naturally results in better investment outcomes. - Oh, me too. I mean, it requires dynamic culture where certain people who wanna go do deals and build their CV, I mean, I hate to be that blunt about it. But if you sit on the sidelines for two years, what do you put on your CV? - Yeah, yeah. Well, and even, and what's awkward about doing these rotations, even though intellectually it sounds like a great idea and practically, and if you look at the results, it is a good idea, but when it happens to you, I remember what I moved from private equity over into fixed income. I mean, those skills don't really port underwriting private equity funds and then trying to trade a liquid fixed income portfolio, which had a bunch of mortgage backed securities and, you know, another types of structure product. All of my contacts in private equity were like, dude, what did you do? What did you do wrong? So you, I mean, it is kind of humbling. It's not easy to do. And people wanna be specialists, like you said, build a CV and it does take a certain kind of personality to get comfortable with that, but it can be awkward. But growth often is. - Growth is. And the funny thing about this is really an apprentice industry. Like I-- - Yeah, great point. - I can find you classes to take on private equity or venture capital at somewhere, some university, but their class to do what you do, it doesn't exist. In fact, I think maybe you took the class or the master's program most aligned with what we do as institutional investors and that is a forecasting toolkit. So with your permission, I'm gonna jump into the case study. - Sure. And that, just for your benefit, is kind of a classic methodology in academia where we try to understand the problem that eventually we're gonna talk about the solution. But let's start with the problem. Then we'll talk about the solution, but really where I wanna like double click is, how did you get the resources? Which is another way of saying cultural buy-in, board buy-in, you know, literal money, in order to actually implement the solution. Then we'll talk about the outcomes, and then we'll let Daniel go deep with his deep thoughts. And then we'll summarize. - Daniel, any commentary before we jump into the case? - I'm psyched about this. (laughing) - Be too, I'm really interested in whether prediction is even possible. So I'm starting from that position of enthusiastic, skeptical, but excited to hear how we can look forward at markets and say something of value. - Beautiful. - And I will say, this is such a neat time to be having this chat, because with the sort of collision of human intuition and algorithmic decision making coming, it's like for the longest time, we've thought of investors almost as true artists, this ability to see patterns and draw pictures that no one else can tell stories. But now we're saying what technology does, and now there's this big confrontation coming. And I think that probably starts to speak to the problem that you spotted Mark, but why don't I turn it over to you? I know you've built this decision making capability. Tell me as you're coming in, before you did that, what did you see, and what problems were you trying to solve? - Yeah, well, I picked up on that last point. I, what I saw in like 2008 and 2009, when we went through the financial crisis, and there was this big sort of uproar about the failure of risk models. And it just seemed to me that we were all looking at the world like it was Newtonian physics, right? That you could explain all the parts and define exactly what was gonna happen. And that sort of, that idea permeated the industry, right? In terms of our risk models, how we build risk models. And what we're really actually dealing with is biology. That's really how the market behaves. We know something today that we didn't know yesterday. We adapt 'cause something happened to us that we either liked or didn't like. What I really felt strongly about was, how do you make better decisions? Just acknowledging the fact that markets adapt than people adapt and there have to be some rules of the road. So the problem, if I'm just kind of following kind of the outline here, the problem is like, I noticed not just our investment team, but our industry, right? Was making forecasts, which is just another way of saying decisions, right? Tons of decisions about markets, economic trends. But because accountability is really bad for career progression, but really necessary for progress, it pays, I think you alluded to this, to be really kind of ambiguous about your forecast, right? Your terminology, some of that comes from compliance. They don't want you to say too much, part of it's like litigation. But I think a lot of it's just job security. And so there's just a fundamental lack. There's just no systematic way to evaluate really the accuracy of these predictions. And so our analysts and PMs would make confident calls, but we couldn't really tell who was actually good at decision making versus who just sort of sounded convincing. And it's usually the loudest voice in the room. And this was particularly frustrating because we're making major asset allocation decisions based on these decisions yet, you know, you look across not just us, the industry, no real accountability or like feedback loop. And that was really the problem that we sought to solve. So what did you build? And I want you to talk about the capability of the internal sort of gamification you built. It's pretty, it's actually not difficult. We implemented really of a formal forecasting system for lack of a better term based on briar scores. And you can read about briar scores, but effectively they go from zero to two, zero meaning you're kind of omniscient and two meaning you're not. So if you're 100% confident and 100% right, you would have a kind of a zero score. And if you're 100% confident and 100% wrong, you have a two. So each analyst or PM at this point now submits their investment recommendations with specific probabilities and time frames. Like, well, say there's a 70% chance the Fed will raise the baseline rate by at least 25 basis points at their next FOMC meeting, right? That is a very specific forecast. Also with investments, you can say, that's an active long-only equity manager. They'll say I'm 60% confident this active manager will outperform say the S&P 500 by at least 100 basis points by 12, 31, 20, 26. And we write that down. So these predictions are tracked. And then they're scored. It's a binary outcome, either you are right or wrong. And that gives us a pretty clear measure of skill across the team. The problem with just looking at the outcomes, which the industry will do, right, is that we know the outcomes are to some extent a factor of luck and skill it. And you can't really tell X-Post, which is, and Daniel's point, there's a lot of revisionist history when things go well. We kind of uncritically accept the good outcomes and we sort of chalk the bad ones up to luck. But if you ask people their predictions ahead of time, right, that's more impressive. If you, for example, asked everybody, I grew up in Michigan and if you had everyone stand up in University of Michigan Stadium, football stadium, and flip a coin. And if you get heads, you sit down and tails, you stand up. Well, pretty soon after 15 or 20 flips, you're going to have two or three people standing. But that's not because they're really good at flipping coins, right? It's just somebody had to be it. But if one of those or two of those, all three of those people told you ahead of time that they were going to do it, that would actually be different. And that might be revelatory of skill. And so that's really what we've built is the system to track recommendations. So we force the team to be very specific about investment recommendations or any material recommendation. And we score it. And those scores are available for everyone to see. Mark, I've got a quick question for you about it. I know I love actually that we're getting right into the weeds. So forgive me for asking a layperson's question about stuff. I don't know anything about. But as you're describing this model of the zero to two ranking and the competence and the outcome, one question I have is, if I were to compare weather prediction to financial market prediction, a big difference is that in predicting the weather, which we're still terrible at, by the way, it said it was supposed to snow 17 inches last night in New York. We got not. I don't know how-- Well, it's pretty easy. They're pretty accurate here. It's good. Well, that's because I was just visiting Mark a few weeks ago. It's awesome where he left. It's just like 72 in sunny and the winter. So anyway, one of the reasons is especially frustrating that we can't even get weather predictions right is that weather predictions do not have to worry about weather predictions. So there's no feedback loop. It doesn't matter if we predict that something's going to happen. That doesn't have an impact on whether it will or will not happen. In markets, the predictions that people make get baked into the pricing, get baked into expectations. If you even think about a simple situation of predicting a Fed rate cut or not, or predicting some sort of employment statistic, so it's self-interacting. The predictions themselves play a role in whether your prediction comes to be true or not. Does that added degree of complexity factor in as you're evaluating somebody's ability? Are they meant to themselves look around and take into account what everybody else is saying when they make their own prediction? Yeah, to some extent, that will play into it. So think of it this way. Max uncertainty would be 50%. And so I'm back to asking just a little bit here. But if you-- it's funny how often you will ask people a question about something they know nothing about. We've all had those questions. How many tennis balls fit into a Boeing airplane? And you give an answer, and so many people will say, they're like, well, how confident are you that you're right? 80%. And that's a really high confidence level. Max uncertainty is 50% because that's a coin flip. And so what you start to learn is that, yeah, everyone's-- your market prices are the result of everyone's predictions. But we're measuring the individual particular. And they should consider that. If I'm predicting this stock or this manager or whatever the FMC, then maybe other people are doing it too. And there's waste. We trained to get better at it. And Ashby brought this up earlier. There are ways to improve your calibration, which is what we're after. So we're not necessarily looking for people to be 100% accurate. What we're looking for is if you say you're 80% confident that those 80% confident decisions that you made are recommendations occur 80% of the time. If you are looking at a recommendation that you have to make, and you are on the fence, it's 50/50, then you're going to behave differently. Ultimately, it gets to investing. What happens is that you start to look at the upside and the downside of the trades differently. I am 100% confident that cutting fees will result in better performance. That makes sense. In the venture capital world. You might be a coin flip on most of the deals that they're looking at, but that might be okay, right? So you have sort of like confidence or sort of improbability or one thing payouts are another, but we so often take them together It's the same thing and our point is that you have to distinguish first of all the probabilities from the payouts So you know first of all make sure that you're Calibrated so that when you say you're 60% confident that those things actually happen 60% of the time, right? Then you got to worry about okay, if I'm right what's the upside and what's the downside? So ultimately, you know, a long way of saying it doesn't matter these predictions kind of feed it That's something that you would want to take into consideration, right? And that might increase or decrease your certainty You know around the outcome, but you've got to consider all of that, right? And I do get a little bit concerned when everyone on the team You know, I'm looking at all their forecasts and let's say they're all pretty high like we had a question about how many Gold medals would the US get right in just a training exercise and Everyone had really high confidence levels about the number of gold medals that we get and I get more concerned when there's Consensus around consensus, right? That's that's part of that's part of the problem and that happens sometimes But yeah, I mean these predictions do feed into market prices and if you're thinking that doesn't mean you're wrong or should make the investment Right, it just has other implications about maybe what the payouts are and and if everybody thinks this what happens if we're all wrong, right? Like interest rate cuts cut interest. You know, everyone thought that there would be you know several rate cuts in 24 Right and that you know that wasn't necessarily the problem is that there was nobody who thought there would be less than like four or five So you see a lot of consensus around the consensus that's that's usually an indication that there's gonna be some volatility if it goes the other way You'd like to see bets on both sides of in terms of like implementing this type of a scoring model for decisions First of all, there's two quick questions. I have one it feels like These questions can be applied to people managing fixed income or venture capital because the the pricing of those outcomes Already is inherently different and you're asking for confidence plus outcome So you kind of take a total portfolio approach with these Scores and the second question which maybe is linked is is it really important how you ask the question? Because if you're asking about a rate cut that's binary if you're asking about how much They're cutting the rates and theory that's no longer binary that's a spectrum can you do both do you need to choose a binary or yeah? Yeah, yeah, it's a great that That's an excellent point the question I find in across the investment landscape is the most important part of this whole Exercise whether we're talking about you know forecasting recommendations or machine learning and AI Everyone talks about how machine learning AI are gonna change the world and it might but if you're not asking the right questions Large language models aren't gonna help you and forecasting is the same way I just happen to prefer binary outcomes So as we look across the asset class because there's so much strategic clarity in doing that So if you look at private equity for example if I'm asking what the return what you think the return is going to be the fund Right, I mean that's gonna I don't know you're 15 to 35 percent. I don't know 50 percent you can put bins around some of those Do you think it'll be between 15 and 20 20 to 25 right and there are ways to score that just because a little more complicated We like to simplify it and say is this going to be a top core tile manager because that's ultimately what we're what we're after if you're hiring an active manager You want to know if they're gonna beat the benchmark you may or may not care by how much And if you do that's a good discussion at the team level to figure out You know what your strategic position is that's why I love the binary You know do we do we if it's 50 basis points is that exciting not really is it 100 basis points not really 200 Yeah 200 okay, so how confident are you that it's gonna outperform the index by at least 200 basis points I'm not it's not a number that can kind of go infinitely up This is an industry. I don't know if you two of you have observed this But there are some big egos in our industry There's some confident humans in this industry and I have a Spish in mark that as you start to reveal even among teammates who is Good at forecasting and who is not good at forecasting there might be some like slight embarrassment So I'm curious who on your team like are you surprised? They're super good? Are the you know superstar portfolio managers? Are they not quite as good? I obviously don't want you to call anybody out and get yourself in trouble But I'm just curious as you get into this and you start to see transparent outcomes like is some of it surprising Yes, it is surprising that that's but that's what I love about this model because it is very egalitarian Everyone scored objectively and that social vindication is looming large and what happens is The first forecast every employee makes you know the join the team I explain this to them and it sort of sounds like a gimmick at first and they don't really you know They don't really put their heart into it or or maybe they try but they just don't quite understand Yeah, how to be well calibrated and they'll come out in their first forecast will be 90% confident 80% confident and they'll get it wrong And you'll see very quickly a self-correcting mechanism And a lot of them will say wow I had no idea I'm this bad at Making decisions because most of us walk around thinking yeah if I ask somebody you know how you know how often are you right? Most of us we actually think we're right like almost 100% of the time and and we and we'll head didn't say 80 And we think that's a big hedge But when you when I'm looking at my team now and I've got you know hundreds of observations But most of our investment recommendations sit between kind of 60 75% you know confidence and it is surprising because What we started to see at the at the beginning were the analysts who were better at it than the PMs We also saw outcomes that were you know The women were better than the men and that was another thing that was surprising And there were some lurking variables there that we had to address some of them were putting more time into it Some of them were busier than the other Than then somebody else right and so You know you we were trying to control for some of these different variables But and some of the decisions some of the forecasts were based purely on machine learning models So this is the way to sort of unify Everyone's biases you have the person on your team who likes to just make the gut forecast And you have the person on the team who likes to just run models for every about everything And if you're not measuring And the person who's making the gut decision just doesn't trust the machines And the person who's made you running the machine learning models doesn't trust the person's gut But if you're scoring them and one of them happens to be a lot more accurate than the other You start to ask questions and you start to break down a lot of the biases And if you're trying to get to truth That's the only way I can think of doing it. You're not going to vaporize everyone's biases But they do need to be controlled And if you see a machine learning model that starts to be super accurate That makes you ask questions like well, why would that be? And if you have someone who's got a reliable gut You know they've got all these heuristics built in from experience You know you can start to kind of tease those out pull them out and and learn more about that and help You know teach the team So we're all about trying to make individuals better forecasters But also collectively make better decisions as a team Let's talk a second about the resources that you need to do this You've already talked about one which is as you're onboarding people You're taking the time in those onboarding meetings to set a culture and explain a process Did you change your investment memos? Did you have to build a new system to accept these bets? Did you have to talk to the board about a change in your culture or your approach to this? I'm just curious about all the resources All the above This is Luckily to some extent resource light If you did nothing else except write down people's investment recommendations And go back and visit those You would see improvement I think in your decision making that by itself Even if you didn't have a scoring system necessarily And the scoring is important because you Even if you're off by fiber You know let's say you think you're 75% accurate But you're only 70 or 65 that actually does have pretty profound investment implications Because all of us know how to structure trades If I'm right I make this if I'm wrong And if you're wrong more than you think you are Your expected outcome could be negative instead of positive So we do score for that reason Our investment memos or our investment team has discretion On every memo The portfolio manager and the analyst Have to deliver their recommendation And they have to We're clear about what success looks like across different types of investments You know so how you know what's your confident that this investment will be successful for private equity fund That might be in the top core tile and beating the Being the benchmark Over a five year period they'll say 70% confident Those go in our minutes and those minutes are delivered to our board We do talk about this with our board I think as a if I were a board member I would be very comfortable with this because they know like this is how we're keeping This is how we're holding our team accountable Fundamentally they are being scored And and it's not the subjective on the one hand on the other hand And that engenders I think a lot of trust between them and us And it's really easy when you're doing performance reviews Or when we have new ideas But for example we had an idea we were running an incubation You know scoring the outcomes of various decisions on paper That we were doing on paper And then using that in real trading And the board is comfortable because there is that process. They know we're scoring. We have a graph that we show where the X axis is predicted and the Y is actual. So if you think about that, you just want to be on that straight line. So your 20% predictions happen 20% of the time, 50% prediction, 50% of the time, 100% 100. And they can see where those decisions are and how closely we hugged that line. And whether we're well calibrated, that's the term we use. So that part is very resource light. It's a spreadsheet to track everyone's recommendations and the outcomes and just calculate the briar score. But what can get expensive is to improve calibration and forecasting is when you bring in the statistical models and machine learning. Now that's not over the expensive because you just need Python code. So to some extent, you can do machine learning with the Python code. But if you start to move into AI and some of these LLMs, that does require resources. The other resource that we don't talk a lot about is just time. So a lot of processes that the investment officers conduct are manual, accessing data rooms, reading PDFs. And so investing in technology to streamline that access documents without them having to do it, indexing important information out of the text documents. So you have a database. That frees them up to then review and consume information. And the more time they have to do that, the less decision fatigue they'll have. And that will also improve decision making. - Daniel, I would have come to you in a second to do your deep thoughts, but first one more question, which is time. So I run the research center on long term investing. And we're talking about bets. Does it matter if we're making five-year bets and inherently like things are gonna just naturally get less confident? And so that's how you calibrate these models. Just talk a little bit about how you do that. - Yeah. - Yeah. Yeah, time, so the time one is tricky. So in theory, if you could make the optimal decision every second of the day, you would wanna do that. If we make out about what we wear, where we go, who we thought, if, you know, let's say there was a way for us to track that. Ideally, if you can make good decisions on a short time, that compounds out over the course of a year to be a lot of, so in other words, you only need, if you made 18, 20 basis points a week, right? That would be substantial when you, that compounds over 52 weeks right out to the year. So you do want to try to make good decisions on a short time period. So when we start, we started training, we had five day windows. And the questions were really hard because I wanted the team to know what uncertainty felt like physically. And so we would ask, where is the SMP going to close above 5500 as of the close of business on Friday? And this is Monday and you have until the end of the day to get your forecast in, right? That's a very, very hard forecast to make. But if you're right about that and it turns out that the group is pretty accurate, then the five day horizons, the one we care about, 'cause making good decisions on a five day basis over the course of a year is gonna add a ton of value. With private equity funds where you have a 14 year window sometimes, right? You know, they keep extending. It's a little more difficult. And what you don't want to do is say, hey, I'm whatever, 70% confident that they'll be successful, whatever that definition is over a 14 year period. Because meantime, you're not gonna get feedback on that on whether you're well calibrated and you're gonna make a bunch of other decisions in the next 14 years without figuring out if the machine making the decisions, meaning your brain is well calibrated. So what we've had to do with some of these longer term investments is chop them up. So we say, hey, in year one, what normally happened? That's the first question of good forecast and you ask yourself, what normally happens? That gets you out of your own head, right? So if I'm a disinterested third party, what normally happens to a PE fund in year one? And when we had this discussion, people were all, well, it's kind of hard to tell. I mean, they raised the money and blah, blah. And so we stopped and said, okay, can we agree that if there's a key man event, there's a departure in the first year that that's bad, yeah, that's bad. Can we agree that if there's an SEC finding or investigation in year one, that's bad, yeah, that's bad, right? Okay, so we've at least, it's not a lot, but you've at least put it, you've at least defined some measure of success in year one. Do they have it, maybe they made one transaction? Maybe they've closed, they're raised all their money, right? What do we expect in year two? Well, they probably have two or three investments, right? And so at each point, you're defining success. So you're getting that feedback sooner than 12 years. - So Nate, I can even see rewriting memos for the definition of what those generic outcomes are and then asking people to do the specific bets. - Very neat. - Yeah, and that means data becomes an issue, data becomes kind of your currency, right? You could you live your own portfolio, and you can say, well, this is what normally happens to our funds, but now we're actively thinking about partnering with larger organizations that have bigger databases and having them inform what normally happens, right? So in forecast, what you ask yourself, you start it with what normally happens. So when we had the S&P 500 questions and they had five days, you would go back and look at what's the average five day return to the S&P 500, right? And that's your baseline. And you would say 55% of the time, it closes above where it was a week ago. And then from there, you start asking questions, well, what would make it different? Well, there might be an election, there might be an economic release, there might be something else going on here, could be a lunar new year. So and that moves you from your 55, either up or down, right? And the same thing with private equity or hedge funds or single stock bets, right? It's, well, what normally happens to us, you know, the stock, what normally happens to a private equity fund in your one or your two? And then why do I think this is going to be different than that? But it's important to think kind of probably ballistically and communicate uncertainty than it is certainty. And I told my, I jokingly told my TV other day, guys, keep in mind that the people, 75% of the people you sit across are not going to be in the top quartile, right? Yeah, of course. Most of the meetings you hold are, like, that's the default, right? That's your baseline is that they're probably not in the top quartile. And you know, that's, I think the more honest you are about what you don't know, the better you are actually at making decisions. Here's a resource that I'm hearing, which is as you're going into these private managers, your side letters or your, whatever you're signing, you should have data requests that meet the data that you need to validate those intermediate milestones. Yeah, that's exactly right. We're working on that. It's tough to do it because some of them, if they give it to us, they've got to give it to other people. But it's also interesting to just tease it out of them. You'd be surprised how many managers have a really hard time defining what success looks like. You can ask a hedge fund manager, what do you, what do you want to aim for? And they'll say, well, kind of a mid-teens return. I'm like, what does that mean? 15% and they'll say, well, between 10% and 20%, over what time period? Short to medium term. What, two, three years? Two years. So you pin them down. And they'll say eventually you get to say 60% of the time. We're looking for a 12% return on about a 12-month horizon. And then we're tracking that. And most of our managers come to find out or overconfident, right? They're 60% confident, but they're 40% right. And they're underperforming. And that's the information you want up front. Because if you don't get it, then when they're in their office is explaining themselves, Daniel's point is just all this revision is history, and you're kind of a sitting duck, because they know way more about their book and the market than you do. So we're big fans of trying to get that information up front. And then updating either through contractual obligations or is in our quarterly conversations. Like, hey, I remember you said that you expected EBITDA over the next 12 months to go to here. You were kind of 75% confident. And we don't need to be that road about it, but we're tracking it. And after we get so many observations, you're kind of like, you guys seem to be really overconfident. And that's important if you're an LP allocating a capital, because you're not necessarily an expert in what they do. But if you're scoring them like this, and you get the definition of success from them ahead of time, you don't have to know what they're doing. You just know they don't know what they're doing, because they wouldn't be that miscalibrated if they really knew what they were doing. And so I think it's important to impose it on your team and impose it on other managers that you allocate to. You know, I'm going to do something I don't normally do, which is I just have another question for Mark before we go into the deep thoughts. I'm just, I'm so curious. OK, Mark. So when you've identified on your team, the folks that are the best decision makers. And I don't mean the best predictors, right? Because it's not that those folks necessarily have a crystal ball. It's that to your point, they are confident. Their self, their confidence in their own ability is commensurate with their own ability. So they know when to lean in and whatnot. So those people that are outliers as great decision makers, how do they, what's their progression to get to that point? Like, are they, one story would be, you're just kind of born that way. And when you said women are better than men. as a rule of thumb on my teeth. That made me think that, you know, there's probably some elements here that they come to you just with a psychology that's closer to that end state that you're going for being a great decision maker. So maybe they were always good at it. The other narrative would be, that would be also very interesting and compelling and kind of believable is, you know, you got to learn from the school of hard knocks. You got to be beaten up a bunch to see just how wrong you were. And so maybe the people that wind up being the best decision makers are the people that screwed up the most, but had the kind of self-knowledge slash gumption grit to stick it out anyway. So I'm curious, maybe it's both types, I don't know, but is there a, can you predict who's going to be a good predictor? - Yeah. You know, it's funny. If you asked me that five years ago, I'd probably would have said no. But now I actually think there are some markers and one of them is humility. And the more honest you are about, again, when you don't know, usually the better you are at making forecasts. And so when we notice like, for some reason, the women, it's not a huge sample size, I mean, they're through a forum, you know, and they happen to be like newer, didn't come from the investment world, right? So they came in eyes wide open and there's been all this research about how experts aren't really accurate about, you know, and experts are good about laying out, you know, what matters and what factors matters, but they're typically not very good at making long-term forecasts. And so what I see on the team, people who have a lot of experience really struggle with this, because they've sort of done things a certain way. It whereas like the younger kind of newer analysts tend to come in what eyes wide open with no preconceived notions of what happened. And those who are willing to just be agnostic and honest and follow the steps, meaning they ask themselves, what normally happens, you know, you can't, I mean, most of us, you know, you know, we want to be, you know, we want to conserve energy, you know, there's a lot of conservation of energy, and mostly our own is what we want to conserve. And so we tend to be kind of lazy thinkers, right? And Dan Coneman has written a lot about that, right? Thinking fast and slow. But you have to ask yourself, what normally happens? And then, you know, that requires some analysis. I mean, you know, you've got to do some, you know, maybe a regression analysis or just, at least look at a time series. And so I find that people who are diligent and just intellectually honest with themselves tend to be the best forecasters. And you can tell it in the way that they communicate. They don't have adjectives and memos, they're not allowed to put adjectives in our memos. You can't say substantial value add. You have to do the value attribution bridge and we'll decide whether it's substantial or not based on the comparisons of other funds, right? But you just can kind of tell the way they communicate that they're just, it's just the facts. They're just stating the facts. And, you know, the more I'm around people like that, you can kind of pick up on it. And yeah, some of them do get beat up pretty quick. I mean, they'll make a bad forecast. That certainly will adjust people too. But I'm finding it's just who are humble and intellectually honest are often the best forecasters. And they think probabilistically. - It just reminds me of my favorite line from a venture capital manager once that came into my office and said that his greatest ability was his humility. - He's the humbleist. - In the world, you know, it's the truly humble, not the stated humble that you want. - So here's my deep thought for you, Mark. This is the part of the show that usually goes off the rails for a minute and then we try to bring it home and our guest is usually giving the tall task of trying to parse through my word salad and come up with a bit of a response. - Now, I just hope you're just gonna read some Jack Candy for you know, from essence. - I got a two and I've got dad jokes. - Yes, we got the whole thing going here. - All right, fire away. - I find it totally credible that you could train a team based on your perspective and training to be better decision makers than teams at other allocators. I think your performance reflects that. I have no doubt that an individual can improve. So at the microcosm, all of this makes sense. I'm wondering about the sort of macrocosm of humanity as a whole. Like, are we getting better at making decisions? Economic policy decisions, financial markets decisions. So there are some reasons I might think that we are. So you mentioned Dan Coniman. I think we've got a better understanding than we did even 10 years ago, let alone 50 years ago, about the kind of inherent biases that the human brain is wired to make, the mistakes that we're wired to make. And that gives you self-knowledge that you have a chance to overcome. So in terms of self-knowledge, hopefully, we're getting a bit sharper because we understand our own cognition. I think our data is certainly getting better and much more available. Our compute is way better, right? So the ability, whether that's AI or just Python or spreadsheet, our ability to actually take the data, combine it with our own views and come up with a prediction. So to the extent data, self-knowledge, plus our understanding of the world matters, you would think we would be doing better. But at the same time, there's this measure every year where they take the Davos consensus. The people that are supposed to be the best at this, with the best data, the most compute at their fingertips. And the Davos consensus is almost always wrong. And you should bet against it. So that would make me wonder whether we actually have improved. The fact that we haven't had a great depression in almost 100 years suggests the opposite. Maybe we have gotten somewhat better at economic policy and we should all be grateful that the Fed and others are able to steward the economy in a way that makes use of better predictive tools than we had in the 1920s and '30s. But I'm honestly not sure of the answer. So I'm curious maybe, if you were to take the under and say we're not getting better, you could argue that just as our skills, our data and our compute have improved, the world has equally gotten more complex and chaotic. And therefore, we've got stronger muscles with the weights who've gotten heavier. Or you could take the over and say, no, we actually have gotten better at it. I'm just curious what you think. Are we better decision makers as people? Yeah, a lot of existential questions in there. I mean, I guess it's asking, are we better off for having developed nuclear weapons? I think every innovation can be used in a bad way. That's, we're clearly economically prosperous. I got turned on to this idea of forecasting by participating in the Good Judgment Project, which was run by Phil Tetlock from 2010 to about 2014. And it was a competition sponsored by the Department of Defense. And they wanted to see if they could kind of crowdsource geopolitical questions. And Phil Tetlock taught us how to make forecasts in the discipline and his team destroyed the other teams from Harvard University of Michigan to where in the second year of that competition, the Department of Defense said, just forget it. We're just going to deal with your team and not the other teams. And so what I learned there is that people can learn to be better forecasters. And you can certainly, you can teach people to be better forecasters. I think what's tricky is people don't always want to be better forecasters. And I think to extent, we certainly have all the tools. We have more tools that you said you mentioned data, which can also be manipulated and have the opposite effect of actually getting people to truth. It can mislead people. And so I find that we certainly have the tools. We certainly are economically progressing. Tough to say whether we're making better decisions, I think on the hold that you could say we are because we're making improvements in various aspects. We're generally kind of wealthier. And we're there's drug development. And we're eliminating diseases and things like that. But I really think it's hard to say. I don't know as the answer, but people have to want to get to truth. And I'm not sure everybody wants to. I think there's a lot of tribalism going on. And people are going to find whatever facts fit the data. Whatever they want to argue. So that's a tough one. I do think social media and the data does make it-- I think it does speed up the outcome. I mean, it does make the outcomes obvious and it's faster. But I'm not quite sure everybody really is intellectually agnostic. I think a lot of them want to light up with specific groups, interest groups. That's just human nature. I love that. I think you took my deep thought and made it deeper. I was wondering about whether the world was becoming more chaotic at the same pace that we were becoming better decision makers. And you raised me and said, well, maybe we don't want to be better. Like maybe there's a nihilistic impulse in humanity too to just tear, tear shit down. And I mean, we're seeing a lot of that right now. So I hear you. Yeah. Well, and keep in mind that like what you have to fight against when you're making forecasts or are the biases, right, that Dan Cannonman and Dan Ariel, all these guys right about, you've got these emotional cognitive biases. And those are strong. And at a very fundamental level, we want to survive. We want to survive and we want to have our tribe, right, and we want to be accepted by a group. And those are very, very powerful motivators. So you do have to actively kind of fight against those. And hopefully, there's some given taking at the aggregate level, we make some improvements, but that will be fits and starts. But the attitude has to be there. Certainly, we have the tools, but I'm not sure the psychology is there yet. Amazing. So I know we're getting near time and I just want to summarize a little bit. I think what we're talking about is our signals are getting better, but so is the noise ramping. And so as the noise goes beyond what we can control, it's difficult to know if our signals are actually getting better, but they are, right? Like we can see the signals, we can tie them. You know, when you take the knowledge out of chat GPT and you start applying it in your life, you can quickly see that like you can make better decisions. It just so happens that there's so much noise now. Well, yeah, I think it was Dickens, right? It was the best of times. It was the worst of times. Age of wisdom, age of foolishness. We had everything and nothing. Exactly. And that's now. That's now. And so to build on what you're doing, you have spotted this problem where it's our job to make predictions and to plan the future. And it's funny. This has been such a fun discussions, which why it went long. Because when people ask me what I do, I say, look, I'm actually trying to improve investment decision making. That's at the core of my life governance, organizational design, data, et cetera. And so in your fund, you have actually built a very specific project around doing that. And it's about predictions and your confidence level around those predictions. And you can track this and you can score people. And the exciting part from my perspective is it doesn't sound like you need a huge amount of resources to get going down this track. You said, right stuff down. That's like pretty easy to do. Then it's like, yes, building a culture or socializing as people on board, changing your memos, talking to your board, building a language around this work is very powerful. But then you start really learning how to be better thinkers, chop up decisions into smaller pieces, be humble, learn how to ask questions. And ultimately, if you do all this right, you get better decisions. And in our world, that means higher returns. Would you change any of that summary, Mark? Nope, that's a perfect summary. Beautiful. Yep. With that in mind, Daniel, thank you, Mark. Thank you. This has been one of my favorite discussions. And we'll be back with more awesome case studies. So great. I learned a lot. Thank you, Mark. Yeah, thank you, guys.