Causal Inference & the "Bayesian-Frequentist War" | Richard Hahn S2E8 | CausalBanditsPodcast.com

The thing I love about conformal inference to the extent that I know anything about it is finite sample guarantees are everything. My problem that I have with his classical frequentist analysis is not that it's frequentist, it's that it's asymptotic. And in my experience, I've had situations where you think you have a mathematical guarantee and then you run a little simulation study and it just doesn't work the way it's supposed to because the asymptotics haven't kicked in. And the conformal inference gets around that like in a very direct transparent way. And it's constructed in a very beautiful elegant way so that it gives you finite sample frequentist inference. Where I come from, if you can get finite sample frequentist inference, you should go for it. The whole game here is that we wanna learn about our data but we have to keep our mind open to the possibility that we're gonna be wrong sometimes. But you don't wanna be wrong too often. So you kinda wanna be right on average. And to me, that's what a statistician or a modern data scientist machine learning is sort of all about. It's like being as right as possible on average. But the rub there is what average. Like what average is it that you're talking about when you say right on average? The frequentist notion, as I explain it, is that you want to be right on average averaging over possible data sets. So the world is generating data sets for you to analyze and the thought experiment is I'm gonna hold fixed the process but I'm going to hypothetically imagine a bunch of different data sets. And I want it to be that however I analyze those data sets, I do a good job on average over the different possible data sets. That to me is the frequentist thing. The Bayesian thing says, well, I'm not really gonna be analyzing one problem. I'm gonna be analyzing a bunch of problems and the method that works well on one problem might not work well on another problem. So really what I would like is a method that does well on average where the average is over both data sets and also problems. I think the reason people get the impression that heterogeneous treatment effects don't exist is because they use convenient samples in terms of the features that they record. So they get data and their features are things like easy stuff, height, weight, ethnicity, gender, whatever, zip code, whatever's kind of available, right? There's this really bad culture in data science broadly of just thinking about the features as being handed to us by the system. And that's not a good way to do science. When you do science, you don't just take the things that are handed to you and hope that you find heterogeneity in those things. What are your thoughts about AB tests or RCTs and the idea of generalization in this context? So I think about if we did an RCT and we said, what is the health impact of a vitamin C supplement? The answer to the ATE, the average treatment effect would be almost nothing because most people these days have adequate vitamin C. But it's not because it's not biologically active and it's not because it doesn't do anything. It's just because most people don't need it. If you find people that do need it, it's a really powerful intervention. So in other words, if you did your RCT on the general population, you'd get one answer. If you did your RCT on a population of people that were on a seafaring vessel in 1750, it would be a totally different thing. And we do this all the time with drugs. Hey, causal bandits. Welcome to the second season of the causal bandits podcast. The best podcast on causality and AI on the internet. He got into causal inference at a business school and caught the bug for good. Conformal methods are his beacon of frequences hope. He develops new statistical methods at the intersection of social sciences, computer science, and medicine. Yes, professor of statistics at ASU and co-creator of Stocktree. Ladies and gentlemen, please welcome Professor Richard Hahn. Let me pass it to your host, Alex Molak. Welcome to the podcast Richard. Thanks for having me. I'm glad to be here. There's a war going on in statistics for decades between so-called basins and so-called frequencies. And in recent years, there's another group of people who like to use conformal prediction, conformalized methods. And to my best knowledge, at least some of these people don't consider themselves neither Bayesian nor frequentist. Where do you see yourself in this battlefield? I mean, calling it a war is a bit much, I would hope. I understand the point that you're trying to get across. I personally think there's a lot of room for compromise between these groups and I certainly don't want to think of them as fighting, although, you know, they get snarky with each other occasionally. So I was trained as a Bayesian. I went to grad school at Duke and was conscripted to user analogy. And then it subsequently worked with frequentist econometrisions. So I've dabbled in a little bit of these various things, but I would call myself a Bayesian. And the flavor of Bayesian that I am is that I'm a big proponent of Bayes risk as sort of the right criteria. And what I mean by that is we want to think about how our procedures work averaging over data sets, but we also want to think about how they work averaging over data generating processes. And the reason that I feel this is that we have theoretical knowledge about our data generating processes if we're doing our job right, and we should take that into account. So I'm using prior information in an intelligent way. The Bayesian framework is nice for that and explicitly thinking about averaging over the performance across uses as opposed to just over possible data sets. There's a lot of different opinions on this. I'm really focused on what works in practice, and I think that that's a profitable way to go. The conformal stuff is really interesting and we've talked about this before. I came across it first when I was trying to reconcile how Bayesian and frequentist stuff is related and I was interested in prediction. I've heard about it around 2010 maybe. At the time felt like I knew a secret that nobody knew about it wasn't super popular then, although, of course, there was already a book published. It's interesting though, I always think about it as a frequentist method because the guarantees that they offer are frequentist guarantees. And as a practical matter, the thing that I like about conformal inference is that you can put in a base model as your basic prediction and conformalize it. And so if you have a really nice Bayesian model that has imported all of your prior information and yet you have an audience or colleagues that demand frequentist guarantees, it has always seemed like a nice marriage to me. Out of curiosity, if you don't mind me asking some questions, I've not heard that conformalists don't self-identify as frequentist. What do you know what the story there is or do you have specific people in mind? - I don't call names here. - Yeah, yeah, fair. - But yeah, so one of the argumentations I heard regarding, putting this demarcation line between the frequentist and conformal is that perhaps the conformal paradigm is maybe it requires less assumptions sometimes than or less stringent assumptions, sometimes then classical frequentist approaches, so maybe extradibility rather than IID and so on and so on. But if it's a good argument, well, I could probably argue with that. - Yeah, so what do you would say? - Yeah, so I mean, I have something very specific to say about that, which is that I tend not to think about these philosophies as being about methods. I think about them as being about methods of evaluation. So you have a method, the method is your procedure for learning about data. You want to then say something about how good that method is and there are various ways you can do that. One way you can do that is you can evaluate it according to Bayes' risk or you can do it according to frequentist risk. And I think that would be true if the method were a conformal method. So if you think about it in terms of methods of evaluation, like what are the criteria that you're judging it by? I think that conformal inference is clearly a frequentist method because the sorts of guarantees that they get are averaging over data sets for any parameter value. Whereas the Bayesian thing is averaging over data sets and averaging over parameter value. So that the IID distinction or the do you have to specify a likelihood distinction. And this is something I learned from the econometrisions that I worked with because trained as a statistician and as a Bayesian statistician, the straw man frequentist for me was maximum likelihood estimation where you were biting the bullet on specifying a likelihood and you were getting asymptotic guarantees about central limit theorems for your parameter estimates or something like that. And then I went and worked with the econometrisions and they considered themselves hardcore frequentists. They were not specifying likelihoods. They were not doing maximum likelihood estimation. So that was interesting. It opened my eyes to like there's a bunch of different kinds of Bayesians, there's a bunch of different kinds of frequentists. And then yeah, the conformal thing is a new thing, but I think the criteria there is still a frequentist one. I'm going to be able to confite me about this. I will say that the thing I love about conformal inference to the extent that I know anything about it is finite sample guarantees are everything. My problem that I have with his classical frequentist analysis is not that it's frequentist. It's that it's asymptotic. And the conformal inference gets around that in a very direct, transparent way. I don't know, where I come from. If you can get finite sample frequentist inference, you should go for it. Like that seems just like a good thing to have. It's not the only thing, but it seems like a good thing. I don't know, did that answer your question? I think it answers the question. And it also opens some door. I want to get back to one of the notions you used in your answer, the notion of the Bayesian risk. For those of people in our audience who are not familiar with this concept, how would you explain it to them? Maybe even assuming they don't have a very strong statistical background. When you're trying to learn from data that is coming at you at random, or some stochasticity, it's noisy, it's not. The patterns aren't perfect. They're statistical patterns. What it means is that you're not going to be able to get the right answer every time. This is what I tell my undergrad students. The whole game here is that we want to learn about our data, but we have to keep our mind open to the possibility that we're going to be wrong sometimes. But you don't want to be wrong too often. So you kind of want to be right on average. What average is it that you're talking about when you say right on average? The frequentest notion, as I explain it, is that you want to be right on average averaging over possible data sets. So the world is generating data sets for you to analyze. And the thought experiment is I'm going to hold fixed the process, but I'm going to hypothetically imagine a bunch of different data sets. That to me is the frequentest thing. The Bayesian thing says, well, I'm not really going to be analyzing one problem. I'm going to be analyzing a bunch of problems. And the method that works well on one problem might not work well on another problem. So really what I would like is a method that does well on average where the average is over both data sets and also problems, different data generating processes. And what this allows you to do is it allows you to make a method that takes into account limitations that you're willing to impose on the data generating process. So if you're doing stock prediction, and you don't think that the stock market is ever going to go up 300% in a day, you don't need your method to accommodate that. And you can build a method that doesn't allow that. And in doing so, you can do better on the data sets that you expect to get. The frequentest thing says we want our guarantees to be phrased in terms of for any data generating mechanism, even if it's one that we don't think is plausible. So in the one sense, the frequentest thing is stronger, because it works for any under, like the guarantees are relative, the average is a narrower average. If you can get that, then that's great. Sometimes you can't get that. And in particular, you can get maybe better guarantees if you try to do well on average averaging over scenarios. So I guess that's the idea. I think data sets on the one hand scenarios on the other hand. And Bayes' risk is a way of looking at the performance averaging over both things. Whereas the frequentest say, I want to average over the data sets for any particular scenario. I don't know that that's going to help a non-technical audience. Exactly. I tried to do it in non-technical terms, right? But you kind of do have to be used at thinking about these things. But anyway, that to me is kind of the game. And so when it comes down to your initial question of am I Bayesian or frequent assert-- I'm Bayesian to the extent that if I had to pick, I would pick the Bayesian criteria, the one that considers the scenarios as well. But of course, if I can get a frequentist guarantee, that's great. And conformal inference, for example, offers that. So I think that's great. I guess the problem is that people's take on this very often, in my-- this is not a technical opinion. This is a sociological opinion or whatever. But the things that people gravitate towards have more to do with where they went to school and where they grew up and sort of who their friends are, then it does the actual merits of the-- there are very few people that are both dogmatic in their opinions and also have deeply considered all of the options, right? What you have in a lot of is people that are dogmatic, but haven't deeply considered them, or people that have deeply considered them and have very nuanced opinions about what thing that you should use. So I'm not saying that the principal dogmatists are not out there, but they are the minority, for sure. One of the methods that you can encounter pretty frequently in your paper is Bart. So Bayesian additive regression trees. What's special about Bart? What's special about Bart is that it works. So what you're impressed by depends a lot from where you come from. But where I was coming from was 2010 Duke Bayesian non-parametrics. So Chinese restaurant processes, to reachly process mixture models, Gaussian processes. These were the machine learning tools that people were using. And I had experienced coding them up and implementing them and trying them out. And I had done some internships. And I had tried these things. And they didn't work because they were computationally really fickle, meaning that the computation was just hard to get done, right? And also, it was hard to specify the prior in a reasonable way. So that's one thing. If you're going to go to Bayesian route and you're going to care about Bayes' risk, like I said, you still have to do a good job at honestly representing what you think your data generating processes are going to look like. You have to be kind of earnest about that. And in Bayesian non-parametrics, it's very hard to be earnest about that because you have to put a prior over a super high dimensional space. It's kind of just a mess. So then anyway, I started my job. I was doing some applied projects. And I tried this thing that I heard about that my colleague, Rob, had invented. And it just worked out of the box. Like when you do these little simulation exercises where you generate some data and then you feed it to the method and you see if it recovers the truth. It was recovering the functions, the nonlinear regression functions just every, like it was just doing a great job. So it sort of just worked in the same way that we think of OLS is just working, right? It was just sturdy. It's a weird way to describe a stats method, but it was sturdy. Like it felt like it had a good build quality or something like that. It was just high quality. In any way, I mean, I was so sort of impressed by it that I spent 10 years now working with it and kind of trying to poke it and prod it and try to understand this machine, almost from a reverse engineering perspective. Because I know the guys that invented it and I'm friends with some of them. And they thought it was a good idea. They didn't know as going to work as well as it did. So yeah, it's an incredible thing. So let me tell you a little bit about what it is for those of you that it has a name, Bayesian additive regression tree. So what is that? I mean, it's effectively it's a decision tree or regression tree. You've got binary splits. It partitions a covariate space into regions where you can make predictions. So if you're familiar with cart classification and regression trees, that's the engine behind all of this. You have a tree. People like trees because trees are interpretable, they say. But they don't work that well all the time because it's very coarse. You can, if you're trying to approximate a smooth function with a cart tree, it's going to be jagged. So there are various ways to ensemble this. Random forest being a very famous one. Random forest has been around a long time. And random forest, I think, was one of the first methods that, quote, just worked in the way that I said that Bart did. It was one of these things. It was fancier than linear regression. It didn't specify a likelihood. But people found that it really worked quite well. And for a long time in industry, that was the thing. Subsequently, there have been other methods in boosted regression trees. So gradient boosting for regression trees came along. And these methods, what they're different from cart is that there are sums or averages of regression trees where they essentially superimpose them. I wish I can't do a picture. This is not a talk. But if you take two regression trees and superimpose them, you just get a finer tree, basically. And so you can get smoother approximations. You lose the interpretability, though. So one thing that cracks me up is people say that things like XGBoost are interpretable. And I'm like, they're not that interpretable, I guess. Any one tree is interpretable. But if you're taking a sum of 200 or 500 trees or whatever, you lose that interpretably real fast. But the trade-off is that you're able to approximate a richer class of functions. And of course, XGBoost has been, I would say, the go-to, at least in caggle contests and stuff like that for many years. So anyway, when they started the BART paper, that the idea was, well, let's try to be Bayesian about that in some sense. And let's have a tree ensemble where we can do Bayesian inference over the space of different trees. And so they came up with a very clever prior over the space of trees, which is kind of this elegant thing. And then they developed an MCMC algorithm to fit it. And I guess, despite all of what you might think, the MCMC for it works amazingly well. Because the problem is highly over-parameterized, much like neural networks are. There's just so many parameters that there's lots of different parameters that will give you approximately the same model fit. So inferring an exact tree is not going to happen. But getting some collection of trees that gets you close to what the conditional mean is or whatever you're trying to estimate, that works really well. And it mixes fine and the computations relatively fast. And anyway, what's interesting and what I've found out and you can read a bunch of papers is that on a lot of problems, it works better than those non-based Bayesian methods. There are details in how it's computed. There's details in how it's fit. But all of those things, it kind of leads to this magic concoction where it has a very favorable bias variance trade-off on the sorts of problems that I look at, which is that it seems to be able to pick up signal when it's there. And it seems to be able to ignore signal where it's not. And we don't fully understand it. I mean, I've tried. I've made efforts, but it's very much in the spirit of XG boost and neural networks where it kind of works better than we have any reason to understand why. And I guess the reason that I like it, what is its main advantage over XG boost? So why am I not doing that if I think that is so great? And the main advantage is that it's easier to tune. So when you put a prior down on these things, taking this spirit of Bayes' risk, it's really easy to calibrate what you expect the function to look like in terms of point-wise standard deviations or complexity, how bumpy you want the function to look. The big advantage is that you don't really have to do cross-validation. You do a simulation exercise where you pre-tune to sort of tell it what you think you're expecting. And as long as you're not too far away in those assumptions, it will adapt to it. And you don't have to do the data splitting and the cross-validation and the hyperparameter tuning. So per iteration, it's slower than XG boost by quite a lot. But once you take into account that you don't have to do all of this cross-validation, it ends up being the human labor is actually substantially lower and the clock time is better or the same. So yeah, anyway, I'm still kind of excited about it. And I don't really know how to sell it to people other than to say go and try it because that's how I got into it. I just tried it on a few problems, and I was like, OK, this works. I remember a paper you shared with me recently. That was comparing a bar and a bunch of different methods with conformalized bar and a bunch of different methods. And what I was surprised about that over a broad scope of data sets, barred without anything else, without conformalization, was essentially pretty consistently the best method in this paper. And when it was conformalized, it was just unbeatable. Why do you think that this is the case? It all comes down to, when we do a study like that, OK, so this-- I'm going to ramble, and I apologize, but you're giving me a chance to talk about things that I love to talk about. So when we do a simulation study, like the one that was reported in that paper that you're describing, what we do is we generate data from some process that we know, and then we test the methods on it. And what you're doing when you do that is you're doing a Monte Carlo Bayes risk calculation, because you're generating from different DGPs and then you're generating data from those and then you're testing your method. So in some sense, if that data generating process that describes your simulation study was the exact thing that your prior was, then you would be doing Bayes risk. And then Bayes would be optimal. That's the trivial optimality of Bayes is that if you draw from the prior, the Bayes thing is the best thing. So the surprise there, really, is that in a method like bar, you don't really know-- you don't generate method generally from the bar prior. You generate it from something else that you think is reasonable, but they're close enough. It turns out. And so that's actually what I think is going on. I think whatever data generating processes people tend to concoct live in the bulk of the support of the Bayes of the bar prior. In other words, the bar prior is just a good prior for the sorts of data generating processes that we think we will see and that we tend to generate when we do these studies. And it kind of makes it unbeatable. If they were doing it on purpose, we would call it cheating. Right? I mean, there have been many Bayes papers over the years that do this. They generate data from their prior and their model. And then they fit it. And they compare it to other things. And the Bayes thing does better. Well, it has to. It has to do better. That's what it's designed to do. What's so cool about these other studies and some of the studies that I've been involved in designing is that you don't do that. You basically-- you either have separate people generate the data from fitting the model, which is the cleanest thing, or you just generate data in a way that doesn't make reference to the model. It's a polynomial, or it's a trig function, or whatever. And yeah, at this point, I have many years of trying this. And it's just very hard to generate data that is both reasonable looking. And that bar doesn't do well on. And in fact, in discussion papers and stuff, I'll have people come and say, well, we broke bar. We made it not work. And then you go and look at the data generating process that they used. And you're like, oh, well, the treatment arm has variance that is 5,000 times bigger than the control arm. That's not what the prior says it's going to be. Like, obviously, that's going to be wrong. And not furthermore, it's a silly data generating process. Treatment effects that are exponential in some variable and just shoot off to infinity. And so we run into that a lot. Like, people think that they have defeated the thing. And then you go back and you look at what they did to do that. And you're like, well, that's not reasonable. That's an unreasonable data generating process. What's special about bars for causal inference, specifically? Nothing. Nothing. I mean, my attitude about that, I get into this a lot, right? So I do causal inference. That's why I'm here, I guess. And I do bar. And the question is, is bar related to causal inference in any particular way? And the only way that it's related is that in causal inference, you end up having to estimate conditional expectations a lot. And especially if you're doing heterogeneous treatment effect estimation. And to the extent that you have to do that, bar is my go-to tool for learning conditional expectations because it's just really good at learning functions of covariates. So I would say it's not specifically related. There's a thing that you have to do along the way that bar is very, very good at. And as a matter of fact, in specific cases, if you use bar off the shelf for causal inference, it actually doesn't do what you want. This is not a problem unique to bar, but it's a problem, I would say, that's a general for causal inference, which is that when you're estimating causal estimates, treatment effects, it's a difference, right? It's a difference between what would have happened if they were treated and what would have happened if they were not. So it's not a response surface. It's a difference between two response surfaces. And in finite samples, when you need to estimate a quantity like that, you need your regularization to bear that in mind. And so if you just do causal inference with off-the-shelf tools, it typically won't do that. And that's true of Bart and it's true of causal random force or double machine learning or whatever. The way that you impose your regularization turns out to be critical. So the modification that I and my colleagues did to Bart was to repurpose Bart in a way that we could regularize the treatment effect directly. So we want to be able to say things like, I don't think the treatment effect is going to be bigger than five very often, or something like that, for any covariate value. How can I impose that information on the model? Well, in a Bayesian model, you'd say, oh, just use a prior. But the way that it's designed is not parameterized correctly for that. It's parameterized in terms of the Y1 potential outcome and the Y0 potential outcome, and not their difference. And so all that we did, this is our 2020 paper on Bayesian causal force. It's a Bart model, but it's been reparamatized so that the prior information can be imposed directly on the treatment effect, which seems like a simple thing. It doesn't sound like it's not Earth's chattering theory or anything. But in practice, it makes a huge difference. It makes it so that when you're trying to learn conditional average treatment effects, you don't go crazy. I mean, essentially what happens is if you have a very flexible model and you go to try to learn these things, you'll just overfit. And you'll overfit the treatment effect. And this is why people are so skittish about doing heterogeneous treatment effect estimation because they're like, oh, it's not reliable. It's like, well, you're not using tools that are designed for that. So anyway, I would say Bart itself is actually not causal inference specific. The modification of Bart that we recommend for that context is causal inference specific. And the thing that's specific about it is the regularization. To take an analogy, if some of your readers are more familiar with like physics and form neural networks, that's a case where you have a flexible model, and then you have a penalty term. And the penalty term imposes the regularization in a very specific clever way, right? That's what Bayesian causal force is doing is that it's got the same flexibility in terms of the patterns that it can find in the data. But it's imposing the regularization in a way that shrinks towards homogeneity. So if the data is noisy, it will shrink everybody back towards the estimated AT. And it's just a minor tweak, but it turns out it makes a big difference in practice. Mm-hmm. In your works, you talk a lot about heterogeneity. Yeah. heterogeneity's treatment effects. What brought you to this area, to this little street? I mean, to some extent, it was the bad thing, which is that methodologists get interested in a method for its own sake. And don't think too much about whether or not it's a reasonable scientific problem to focus on. So I mean, the honest answer is that I got into it because I was into nonlinear regression techniques. As I worked on it more, and I've gotten more serious about the applications of this work, it's actually a really interesting question about whether or not heterogeneous treatment effects are sort of feasible. I know a lot of people, colleagues of mine, that have started out in this area and kind of moved away from it, because heterogeneous treatment effects seem, they're just really hard to find, right? Like you, you get a data set, and you need just enormous amounts of data to do it, right? And so I've had a little crisis of conscience, which is like, am I doing this, is this all sort of silly? And I've had to convince myself that it's not, that heterogeneity is a reasonable thing to look at. And I guess I'm in the process of it. I still don't know. I haven't answered it for sure. But what strikes me is that we all know that there's heterogeneity. We can all come up with examples of heterogeneity in our daily life that are very, very obvious. And yet, when it comes to our methods, we think that they're almost impossible to find often. And give me two or three examples. Yeah, so one of the examples I'm really interested these days in heterogeneous effects for drugs, for treatment effects of pharmaceuticals. And that's a perfect example where people say, let's just focus on the ATE, let's bring the drug to market. Whether or not it works better for particular demographics or particular age groups or genetic types or whatever, it's always going to be underpowered. We're never going to be able to tell for sure. Meanwhile, I can come up with examples like side effects where we acknowledge that obviously people react differently. Allergies, some people are allergic to penicillin and some people are not. We don't say, oh, we could never figure that out. Like, we can obviously figure that out. And in some ways, the key example is just diagnosis. So I use this example because it's such a famous example in causal inference and in medicine. Scurvy is a disease that is a vitamin C deficiency. We didn't know that originally, but we know that now. And if you think about that, what that says is that vitamin C can save your life if you don't have enough of a few, if you have an adequate amount, then taking a supplement for vitamin C is not going to do anything. So I think about, if we did an RCT and we said, what is the health impact of a vitamin C supplement? The answer to the ATE, the average treatment effect, would be almost nothing because most people these days have adequate vitamin C. We have treatment, the recruitment protocols basically say, we are recruiting people to have a particular condition and the drug is to treat that condition. So the clearest, most trivial example of heterogeneity is diagnosis. If you have the disease for which the drug is supposed to treat you, that is the subpopulation where you're going to expect a treatment effect and in a healthy population, you wouldn't expect one. Now, I have talked to people in pharmaceuticals and I've given this example and they're like, that's a ridiculous example. Like everybody knows that. That's not what we mean when we say heterogeneity. When we say heterogeneity, we mean based on a genotype or based on a blood draw or something more exotic. And I guess my point is that philosophically, I'm just saying there's probably a middle ground between that example and the example where everybody has exactly the same treatment effect. Like there's gotta be something. And so an example of that middle ground is that there are some drugs that are metabolized by an enzyme and some people make that enzyme and some people don't make that enzyme. And for those people, the drug will act completely differently. So I think there are clear examples and I think the trick to it is all in what covariates are measured. What you do is you come up with ideas about what might matter and you go and measure those things. And so, yeah, I guess the now that I've talked myself into a reasonable position, I think the answer is, I do think there's heterogeneity. I think it's an important thing to understand because it's how we understand mechanism. I think that the pessimism about it is because there has been a whole lot of, I don't wanna say laziness exactly, but there's been a whole lot of convenience in terms of what it is that we're measuring. And we haven't really tried to measure. In a lot of contexts that I've seen firsthand, we haven't tried very hard to measure things in an intelligent way. - So we landed the backing sociology again. - Yeah, well, you know, it's a recurring theme in applied machine learning and applied data science, which is that you're trying to do all of this in a system of people. Then you can have a good idea. And if nobody uses it, it doesn't do much. So yeah, I mean, I kind of wish it weren't that way. But I think as a realist, I think it definitely is. I mean, you started out, right? The whole conversation started out with saying that there are these three camps that are sort of against each other, that's a people thing. Those ideas don't dislike each other. - When we started and we talked about frequencies and basions and conformalists, maybe, you said about frequencies looking at the data coming from the same data generating process, different datasets coming from the same data generating process. We've maybe a different structure of noise and not maybe different structure, different amounts of noise and so on and so on. This topic is also very important when we think about our RCTs. You mentioned RCTs, right? We run an RCT in one population, on a sample from one population. Then this RCT will generalize, hopefully, to this population. Assuming that the sample is maybe a random sample or a representative sample, whatever kind of sampling we do there and take into account in our analysis. When I read on the internet sometimes people writing about A, B, T, S or RCTs, I am under an impression that we as a community, we love to forget sometimes that RCTs are also based on assumptions and then their generalizability is also based on assumptions. So for instance, if we imagine the same population, right? So we take a sample, let's say for simplicity, that we just take a random sample from some population. We run an RCT on this sample. We get to some conclusions about the average treatment effect within the sample. And then we typically assume that, well, this effect will now carry out to the general population. But this assumes still that this population will have a constant or a stable or a stationary data generating process over time as well. When you think about crossover trials or a N of 1 trials, this is, people start thinking about this, ideas more explicitly. What are your thoughts about A, B, T, S or RCTs and the idea of generalization in this context? So as a practical matter, the first thing I think of when you say this is that I think that I think about the role of rigor in our conclusions. And this problem of generalizability has always fascinated me because we spend so much effort being rigorous about the sample average treatment effect. And then when it gets to the point where we're asked to generalize, we all of a sudden became, we become very artistic and very kind of like, well, probably it will work. And of course, I'm speaking specifically about medical statistics, people who do RCTs for living. Sort of in that context, there have been people coming out of the causal inference literature from computer science that are particularly interested in generalizability and portability and they have been very rigorous about studying that. It's interesting how different those two bodies of literature look. Like if you look at the way people talk about generalizability from the folks that are running RCTs and then you look at the way that they talk about generalizability in the computer science literature, talking about portability and generalizability, it almost is like two different fields. Well, it is two different fields, but it's almost like totally different problems. So to my knowledge, there hasn't been any hard core applied efforts to think like what practical use could we put rigorous ideas of generalizability? And partly it's just 'cause it's so abstract, it is a very hard problem. I think that's the main thing to repeat myself a little bit, I think this is why I am interested in heterogeneous treatment effects. I think the more you understand heterogeneity in your treatment effects, the better handle you will have on the way that things are going to generalize. You will sort of know who it works for when or at least you can start to answer those questions. So I guess my general thought is I would like to see more regression analyses done ex-post on RCT data, not to draw firm conclusions, but to give us some idea and to motivate follow-up studies. Yeah, I don't know, I'm gonna have to punt on this one. I mean, I know we've talked about this a little bit before. It's an important idea. I don't have really well formed thoughts on what to do about it. Well, okay, so for the first thing, whenever I do my studies, whenever I do my simulation studies or whenever I'm analyzing data, I always make my estimate the population average treatment effect. I just, I prefer to work in that framework and the reason I prefer to work in that framework is that it allows me to think about the thing I actually wanna think about, which is how this is gonna work in the super population. And then you say, oh, but then you end up having to be Bayesian or you end up having to think about these things that you don't get to, you've got people from, particularly from the randomization school of inference, who wanna do sample average treatment effect where the only thing you're modeling is the treatment assignment randomization. They'll say, well, you know, let's be rigorous about that and let's make that link in the chain as strong as possible. And I kind of have the opposite opinion. I'm like, if the thing I care about is the population average treatment effect, let me do my entire analysis with that thing in mind. And if I have to make assumptions about how the sample relates to the real world, let me do it then as opposed to doing it after the fact where I get a really rigorous answer to what's essentially the wrong question. That, I think, is basically where I stand on that. Like, it's not a very concrete proposal, but I do think that there's a philosophical flaw, which is it's the two key quote, like I'd rather have an approximate answer to the right question than an exact answer to the wrong question. I feel like randomization inference is getting a very, very exact answer to the wrong question, whereas focusing on population average effects is getting approximate answers to what is ultimately the question that you care about. You mentioned your studies and in many of your studies, you used simulations. What's special about simulations? - It gives you a sense of finite sample performance. First of all, if you have a frequentest method that is based on asymptotic guarantees and you run it on a study that has n equals a thousand and you're getting a really bad coverage, it means that the asymptotics are not a good guide. So that's the first thing I like. I like to be able to sort of see that. And the nice thing about it is that a simulation study is not a theorem, nobody's saying that they are. I don't think that, but if you have a theorem that says it should work in a certain way, all it takes to disprove that is one counter example and a simulation study can do that. So if you have a simulation study that breaks a frequentest method, that's all you need. Like you know that something is, you know that that guarantee isn't sort of face value. At that point, so that's one thing. The other thing is that it goes back to this idea of Bayes' risk, like when I'm doing that, what I'm essentially doing is I'm doing a Monte Carlo simulation of the Bayes' risk. You can think about it that way. So I think it's actually a principled thing to do. But the other reason that I like it is that it allows me to be, remember I said, to be a good Bayesian, you have to be honest in your modeling. Like none of it works if your modeling doesn't reflect what you really think the world looks like. When you make a Bayesian method or you have a Bayesian procedure, meaning that it's motivated by this idea of a prior. You very often have this thing where the tail wags the dog, I don't know if you've heard that saying, but where the computation dictates your model. So you can only do certain types of things, and so you make your model so that it has, I don't know, conjugate updates or so that it can be put into a stand sampler or whatever, you sort of let convenience be your thing, and you tell yourself that that's okay because I'm just being practical, I have to do that. The problem with that is that you lose all of the elegance of the theory and you haven't really imported the correct information about the real world. So the reason I like simulation studies is that it's much easier to be rigorous and honest and earnest whatever, you know, it's really easy to simulate data in a way that maintains fidelity to the world than it is to build a model that allows inference for that. So in other words, your simulation study can be as exotic as you want in terms of signal to noise ratio and in terms of the number of covariates and in terms of transformations and missing data and like it's pretty easy in a forward direction to build in all of these things about the world that you think should be there. It's much easier to do that than it is to actually build an inference technique that will take those into account. But what you can do is you can take the method that you can do the computation on or you can do the method that your colleagues think you should be using, and then you can simulate the data and you can test it. It's like in other words, all of the things that we think about as points of pride when it comes to modeling, you can really let your freak flag fly on the simulation side. You can make those simulations as exotic as you want and as realistic as you want to or as difficult, right? The other thing is you can do sort of adversarial testing where you make the data really, really, really hard and see what happens. You can make it really, really, really easy and see what happens. So it's part of this idea that I liked, I like to think about these statistical procedures as machines that I want to reverse engineer. One way to reverse engineer is by studying mathematics. Many people do that. There are limitations to that. Now that these models are so complicated, you know? There is a reason that there hasn't been that much theory advances on neural networks because as machines, they're just very, very hard. You can't do a Taylor expansion very well on things that complicated. Somebody's gonna pop up and tell me that they've done this and that you can do it. But in my experience, what I've seen is that they're not that amenable because they're complicated. So in other words, the thing that makes them work is that they're sophisticated and the thing that makes them hard to understand is that they're sophisticated. And you kind of have two options. You can either restrict yourself to things that don't work as well, but are amenable to study or you can work with the things that are sophisticated and seem to work, but give up on studying them. I don't think either of those options are good. And I think simulation studies kind of come to the rescue. If you put in the effort to make these realistic, large-scale, thorough simulation studies, it allows you to understand in a partial way the edge cases that make the methods break, right? So you know where one method works well and where one method doesn't. You know the weaknesses and the strengths relatively. And it's just a way to probe these systems from a kind of an engineering perspective. In some ways, and this is tied to the idea of Bayes' risk, the relation between Bayes' risk and simulation studies, but in terms of what I would say to others, if you're a methodologist and you've made it this far through my randomly, like simulation studies should be taken way more seriously. Because from a publication perspective, people have used them in a really silly way, which is they either make them super easy just to say that they've done it and their method looks good or they've done them in a dishonest way where they use something that is invented to make their method look good. So either cursory or sort of cherry picking, whereas the real power in them is to use them adversarially and to try to break things with your simulation studies. That's where you start to understand the limiting factors of the algorithms. - For those of our listeners who'd like to employ more simulations in their own research or their own practice, maybe industrial practice, what would be your advice? How to build a good simulation? A useful simulation. - Talking is much easier than working. And what I should have, what I should be doing, what I should have is somebody, myself, or somebody more industrious. It would be nice if there were a dedicated package in Python or R to hold the user's hand through this process. It's a difficult package to write because it requires subject matter knowledge. When you design these studies, it needs to be in the context of something so that you can get your scales right. And that said, and it also depends on your estimate, like what are you using this for? A good simulation study is going to depend on what it is that you're trying to learn. But what I will say, and I've said this elsewhere, is that I think of five things. I think of signal to noise ratio. In other words, if I have additive noise or some other noise sort of mechanism, I can make that noise real high or real low. And what you would hope, right, is that methods work better as the noise level goes down and they work worse as the noise level goes up and it's sort of monotonic. Interesting thing that you'll find is that some methods actually bog down when the signal is real strong. So when the noise is real low, because they get stuck in local modes and stuff like that, the search space becomes harder in cases where the problem is technically easier. So that's kind of an interesting thing you can learn when you do stuff like this. So signal to noise ratio. In the case of doing treatment effects, you also want to look at treatment effect magnitude. So you've got variation in your outcomes. How much of that variation is due to the treatment effect? So many simulation studies in the statistical literature on causal inference simulate their data where the treatment effect accounts for like 50% of the variability in the response. And it's bonkers because that's just in real life if you're looking at headache medicine or a blood pressure medicine or like the drug is just not going to have that big of an effect due to other stuff. That's a case where I like to look at that. Like if the thing I'm trying to learn accounts for a bunch of the variability, it should be easy. So it's another type of signal to noise ratio, right? It's signal to noise, but it's signal to noise on the treatment effect as opposed to on the response surface. The other one then is heterogeneity. How much heterogeneity is there? You can have no heterogeneity where you have a homogenous effect. You can have lots of heterogeneity where it's really, really big. Then the number of features that you would like to try to learn, so dimensionality in general. So sample size and number of features that you're trying to learn in these problems. So how many is that? That's sample size, number of features, signal to noise, signal to noise on the other thing and heterogeneity. So those are the five that in the sort of work that I've done over the past five years or so. Those are the things I think about when I do these studies. To get a little in the weeds, all of those things I just talked about, you want them to be relevant at the point where you're doing the estimation. So for example, in a regression discontinuity design, the only thing that is identified is the treatment effect at the discontinuity, okay? What that means is that for your simulation study to be telling you what you want, you need to attend to the signal to noise ratio at the cutoff, the amount of heterogeneity at the cutoff, at the sample size, you know, in a neighborhood of the cutoff. And so that's an interesting thing. You can generate a simulation study that overall, you think you have good control over. And when you then narrow it down to the region of the response surface that is actually identified, those numbers can be quite different from what they are sort of overall. So yeah, that has a practical guidebook that is where I start. You and your team created a package called Stock 3, that implements BART in R and Python. When you put yourself in the shoes of a practitioner, someone in industry working on a heterogeneous treatment effect estimation problem, and maybe they have on their plate BART from Stock 3, some implementation of double machine learning and some implementation of some TMLE-based estimator. How would you guide them to make a decision which one of these three they should pick for their problem? - I don't think they have to pick. So much like conformal inference, TMLE and double machine learning are both sort of meta procedures that can take base learners. So in some sense, the question is, do you want to use Stock 3 as your base learner or do you want to use something else as your base learner? And to answer that, that's a harder question to answer. That's, I'm going to say always because I have a lot of firsthand experience. It's the thing that I trust. It's going to depend on the context. You made a point though. You mentioned that JRS is B-paper that I sent you, which used BART-based learners in a conformalized inference problem and you noticed that the BART thing without conformalization sort of worked pretty well. So that raises the question of, okay, if you're insisting on having some sort of frequent scaffolding where it's conformal or double machine learning or TMLE, do you want to use those in addition to a BART thing out of our package or do you want to just use our thing raw? And I think that if you are willing to do simulation studies and can find out, you can just kind of check to see what gains you're getting from doing the other thing. I would say the main advantage to not using it is that you can often get a good part of the way towards equal performance. And you don't have to spend all the time on the data splitting that is necessary for things like conformal inference or the extra optimization that is necessary for TMLE or something like that. So it becomes a very in context question of how much compute do you want to use to get your inferences. We're in the early stages. So my former student Drew Heron has been working on this really hard. He's done a fantastic job. We're very excited about the package. It doesn't have wide adoption yet because it's brand new. It just only exists recently. But one thing that we need to do moving forward is we need to do some basic comparisons between these other libraries that are offering this stuff. But we're optimistic that it's going to be something that people will want to use. Yeah, I can give you a little bit more of a sales pitch on it. But the short answer is you can use both. You can use them together. And if you want to use them separately, you'll just need to check in some simulations to see whether or not it's necessary to add on the extra layer of inference. We could also conformalize to the double-- Yeah, yeah, no, no, I mean, we've burned by-- So base learner, right? So the reason that I've worked for-- So if I'm looking at those techniques, conformal double machine learning, TMLE, my preference for conformal is because it has finite sample guarantees and the other methods to my knowledge to not. The downside, I guess, of the conformal inference guarantees that most of them are not point wise, they're marginal. Whether or not that's a problem is going to depend on the context. And what I mean by that is, do you want to have a conformal inference that has guarantees at a particular covariate value or across different covariate values, where you could be really off on some and really good on others. Anyway, what they show in that paper incidentally is that even though the guarantees are only marginal, the conditional coverage is actually pretty good, which is this kind of this interesting thing. The actual performance always seems to lag the theory a little bit, but there seems to be stuff that's good about these things that we don't quite understand yet. All right, I'm sorry. I'm talking too much. You got me going. Well, that's your episode, Richard. That's why we're here for to learn from you. When we talked some time ago, you told me about this idea, and that will be related to what we discussed about simulations a little bit, about regimes where things work, where they tend to work. Are there any specific regimes where Bart works really really well and outside of which, it starts to fail? And I won't pass this question in the context of causal inference, precisely because you just read my defects. That's a really good question. So let me expand on the way you phrase that. So what I found in all of these simulation studies that I spend so much my time on, that I keep talking about, what I have found is that if you make the problem easy, everybody gets the right answer. So if the data generating process is homogenous and the signal is strong, and there's not that many covariates on the sample size is large, like that's an easy problem. All of the methods kind of do OK. If the world is really difficult, difficult is small sample size, high noise, high heterogeneity, large number of possible features, nobody does that great. And you can't really pick a winner because they're all bad in their own way, but nobody is kind of getting the right answer. And so taking this as a generality, there's this question, which is how big is the middle ground at all? I think this is-- and I don't have an answer to that question. I think that's a huge-- it's an empirical question. First of all, can't be answered with theory, really. It's just like in the world, how often are you in the super easy case? And how often are you in the impossible case? It might be that the middle ground where it makes a difference if you use Bart versus XGBoost versus double machine learning XGBoost just might never matter. That's depressing to me as a methodologist to think that maybe that's the case, but it's a possibility. Nonetheless, when you're comparing methods, I think you're obligated to focus on that middle ground, because otherwise, what are you doing? Everybody is just doing sort of the same thing. How to characterize that middle ground is not an answer-- like, I don't have an answer to that. I don't know exactly what it is. So what we focus on instead is we focus on this kind of idea. So Andy Gelman, on his blog many years ago, had this quote where he said, statistics is always looking for the biggest effect hiding in plain sight. So in other words, you didn't see it before you did the data analysis, but after you did the data analysis, nobody was like, oh, the p-value is not quite small enough. Everybody's just like, oh, yeah, that's a real effect. So in hindsight, you can see these things, and it seems obvious. And then follow-up studies would definitely show them. He said, that's what we're all looking for. So I think he was talking about this kind of same idea. But I think about that when I'm thinking about heterogeneity. I want it to be the case that if there is a subgroup that has a pretty pronounced difference in the effect, and maybe nobody else does, are we going to be able to find that? And then secondly, are we going to be able to find that in a way that doesn't change the results for everybody else by like biasing it upwards or whatever? And I think Bart does a really good job at that. Bart is able to handle noisy situations, and it does a remarkably good job at finding it when there are subgroups that have a different effect. But it's very hard to characterize. So all of these ideas, by the way, about this signal-to-noise ratio and heterogeneity-- these are unnova ideas. Like these are ideas from the analysis of variance. It's a very old school statistical way to look at it. And it's hard to think about a nova when you fit these crazy models. But when you're simulating, it's pretty easy to think about it. Because this is what I'm talking about. Like it's very easy to control signal-to-noise ratio for subgroups, and how much of the variation overall is described by this factor versus that factor, like using those kinds of decompositions are really helpful. So that said, the flip side of it is, I don't want to get bit by the complexity of my model if the world is actually simple. So if the world is actually a linear model with homoscodastic noise and constant treatment effect, I need to be able to do well. So typically, that's kind of how I benchmark things. It's like, I'll look at a case that's real easy, and make sure that I do OK. And then I'll look at a case that's really hard, and make sure that I don't overfit dramatically. And then I'll look at a medium case and spend a lot of time there thinking about, can I recover things approximately? But I think that this type of robustness is what we try to focus on. So there's this big debate. So I'm on LinkedIn too much, right? That's how we connected. And I spend too much time there. But one thing that you'll see a lot of is you'll see people out there that are like, oh, turns out generalized linear models are all you need. Like 99% of the time, all you need is OLS. Like this sort of like, and I say this in a non disparaging way, but like rank and file data scientists, people that analyze data that do the real work. Many of them have come to this conclusion that the only thing that works is really simple stuff. And I worry that that's an illusion because of this regime issue. I worry that we're in the hard case so often that it appears like only the very simple things work, just because of bias variance straight off. So what I'm looking for is I'm looking for a model that is complicated enough that when something is complicated, it can adjust. But when things are super noisy, it does about as well as OLS or a GLM or something like that. And Bart does really well at that. So what I'm thinking about is like 90% of the time or more we're in this really hard world where really simple models work fine. I want my method to work okay in those cases, right? And I think Bart does a great job at that. XGBoost does not in my experience. It's very hard to calibrate XGBoost to get it to where it ignores stuff, which is what makes it so good at finding all these patterns. But Bart does really well at sort of shrinking everything back towards a constant and fitting a simple line or whatever. But then when it is complicated, it can still do the job. So this is the kind of profile that I'm looking for. And it's what I think Bart is really good at. If you can get that profile, what it means is that you can always use the same tool. It means that you don't have to do this discretionary thing on a case-by-case basis of saying, is the world too complicated and I should just use OLS? Or is it really sophisticated and I should use double machine learning? Like I don't want to have to make that judgment with a pre-test or something like that. I want to have a method that I can use in both cases and they'll give me reasonable answers in both cases. Which goes back to the Bayes Risk idea. I want my method to sort of work while on average across these different cases. How much can we learn about the world? There are limits, right? There are limits. One of the things that troubles me about the RCT or bust mentality that is evidence-based medicine in 2025 is that we've focused a lot on survival studies. And so if you want to know if your cholesterol drug works, you do a large trial that has 25,000 people in it and you look at the survival rate and you do a survival curve and you try to figure it out. And the thing that makes that hard is that if survival is your endpoint, you're not going to get to do experiments, right? Like you can't do individual-level experiments. So by definition, in a survival-based study, you can't do an N of one study. So we talked about N of one or really like this idea that there are some causal inference designs that are longitudinal where you give somebody a drug and then you take the drug away and then you give them the drug and then you take it away. And as their behavior changes and given some assumptions about things not changing over time, you can learn, this is how we figure out if people are allergic. We expose them, they get a rash, we take it away, the rash goes away, we give it to them, they get a rash. But like survival analysis, you can't do that, obviously, because you study a subject and then the subject dies or has the terminal event or whatever it is. And so what I'm trying to say is that we obviously can't answer the question, for you, what is going to make you live the longest? We just can't, we can't answer that because you've got one life to live. It's the canes quote and the long-run raw dead. Like we don't have enough runway to do the sorts of experiments to answer certain questions. And so yeah, they're lots of limits. I said this to somebody, yes, so the autism thing came out yesterday where Trump was saying that Tylenol was causing autism or something and people were talking about it. And one person said, just a reminder, we can't estimate individual treatment effects. Point being, we can't tell for any given person if Tylenol caused their thing. And I was like, yeah, I guess, but that's not the right attitude. Like we do these end-of-one type designs all the time, we learn things and the person came back to me and said, well, yeah, of course, like I turn my light switch on and the light goes on and I turn my light switch off and it goes off and I turn it on. And he said, but technically this is assuming treatments don't change in time. And I was like, yeah, like yeah, it is, but like we make those sort of assumptions all the time. And one day that light bulb would burn out and the assumption will be violated, but it doesn't mean it was a bad way to study the world. So anyway, yeah, there are limitations. Causal inference in general requires assumptions. I guess coming from that perspective, this idea about limitation seems very obvious to me. If you don't have enough deconfounders, like if you don't have enough control variables, you can't deconfound things. You can't learn causal effects. RCTs are a good way to get around that and to sort of fix that. But then you've got the generalizability issue and you have to make assumptions about that. And then in a one study you've got time varying, if things are time varying, it's going to be very hard to tell whether or not it was the treatment versus whether or not it was something else going on in the background. This is the thing I love about causal inference though. Like causal inference as a field forces people to think about what information is coming from the world and what information is coming from an assumption. And it's very upfront about the fact that to make use of the data from the world, we have to make some kind of assumption. So it's almost like your question was, what can we learn from the world? In some sense the answer is nothing unless you're willing to make some assumptions. So we shouldn't be scared of those, right? Like making assumptions is not the problem. Being explicit and honest about our assumptions is sort of the important thing. Richard, what are two books that change your life? Oh wow, that's hard. Did we talk about this before? I've read-- I like to read. OK, I'll pick one to start. My first year of college, I was living in a crummy little dorm room at Columbia University. I read a book, A Biography of Ludwig Wittgenstein called The Duty of Genius by a very talented biographer named Ray Monk. And it was all about-- so Wittgenstein is a philosopher. I didn't know anything about his work at the time I read the biography. I think I got the biography because I had a cool title, like a cool cover and title. But his life was just so interesting. And where the title comes from is the idea that this guy was just compulsively thinking. So duty of genius makes it sound like he thought that he was a genius and that he should therefore use his brain to solve whatever. But that's not in context. That's not what it meant. What it was about was like, this is just who he was. And it's just what he did. He couldn't stop himself from doing it. And he had a big influence. He's a very influential philosopher. But whether or not that's a good thing, I don't know. But anyway, it resonated with me. And as a first year college student, it gave me an identity that I still have a little bit today, which is just like, this is just a way of being. Just like asking questions, being curious about things, trying to figure things out can be like an artistic calling. I had never seen that idea before, really. And certainly not in the context of philosophy. I think that's what I like. This image of artists being like this, I think is out there, right? The starving artist, or the poet, or whatever. I'd never seen it applied to like an analytic philosopher before, or a mathematician, somebody who did things logically. I really liked that. I still, I think about my job as a kind of a creative endeavor more than a technical one. So that definitely, I think, had a big impact on me. Don't I have to pick a second one? I can, I can try. I mean, yeah, what else was there anything more recently? You know what? I won't pick a specific one, but I'll pick a genre. So because I did pick this as my career, and I've been fortunate enough, and I get to think about puzzles all day, it's hard for me to decompress. Because I'm always thinking about these things, and it's my main hobby. Like when you're a hobby is your job, it's a good thing, right? But it's also a little exhausting. And as I've gotten older, what I have discovered is procedural mystery novels. They scratch that itch of being like puzzles, but they're also sort of fun and mindless. And so in the last several years, I've been reading a ton of translated Japanese detective novels. And they're just really fun. Before we close today, what's your message to the causal community? My message is the Oscar Nuri. The causal community, I think, is bigger than we think it is. So that's without saying that Kumbaya, everybody should get along or whatever. What I will just point out is that as you research it, it gets bigger, right? Like all the people doing RCTs in Pharma are interested in causal inference. All the people doing AV tests in online experiments are doing causal inference. All the people that are doing portability studies and causal diagrams, that's causal inference, econometrisions, policy evaluation. I met one guy, and it's funny. He gets super annoyed about people calling it causal inference. Because he's like, everybody wants to do causal inference. Like, why is that a separate field? That should just be called science. And if you're doing statistics in the name of science, it should just be called that. And I mean, I don't go quite that far because I understand, in particular, there are cases for predictive models and other stuff. But anyway, once you realize how big it is, there needs to be more cross-pollination than there has been. It's starting now, and it's good. And people in my generation, I think, have made some headway in that sort of one or two generations removed from the people that originated many of these things. It's starting to filter through. But I still think there's a big gap between people that study things from the traditional RCT perspective and people that do causal diagrams. Like, it's a different lingo and it's hard for people to talk about. The assumptions are out there. So it's not a message, really. But it's a wish, I guess, is that I wish that there would be more openness to these crossover tools. That said, I think the only way to do that is that I think that the theory folks need to touch data occasionally. So these days, there's that saying touch grass. Like, you have to go outside occasionally, get off your phone. But touch data is what I would say to the causal inference. I don't do it enough. I mean, do as I say, and not as I do. I mean, nips is a good example-- sorry, I'm old. Neurips is a little bit of a good example of this. Like, it's a community that does really interest in causal inference stuff. But the number of people that have data sites in their papers is just shockingly low. And that's a problem for a methods community, right, to have a methods that are evolving, sort of not motivated by primary problems. Yeah, if you do causal inference, try getting new ideas based on data. Well, you have a data set, and it has a problem. So recently, I got into survival analysis kind of like this, like I think causal inference, heterogeneous treatment effects for survival curves is one of the projects I'm working on right now. I'm not the first. There's a few other people that have looked at this. But that was a case where somebody gave me a data set. There was this study called the Vital Study, which was looking at the effect of omega-3 supplements on cardiovascular health. And they had a big randomized trial. And there was some interest in looking at racial disparities in the response to the treatment. So hey, I'm that guy. Like, I've got software that can do heterogeneous treatment effect estimation. What can you do? And I immediately was like, I don't know how to handle survival curves, which is the right way to analyze that data. And that happens more often than I would like to admit. And I think that more of my brilliant colleagues that are doing theory stuff should have this experience. I'm being like, here's a data set, do your thing, and then like, oh, I don't know how. And it's true with software too. We've started building this software partly for this reason, because it's like, if you want somebody to use a method that you think is a good method, and then they hand you a data set, sometimes there is a big gap between having the data in your hand and being able to get an answer out on the back end. And building software is one way to make it so that that can happen. But yeah, I mean, in a way, I'm just talking about the theory divide, the theory practice divide that has sort of always been there. But I think that the only thing in my experience that gets people motivated about closing that gap is feeling the pain of trying to apply some fancy method to a particular data set and not being able to. Yeah, touch data occasionally. That's my essentially. What question would you like to ask me? I want to know what you think you have learned or are hoping to learn from these conversations that you're having with people. You have talked to an impressive-- I mean, I'm very humbled to be talking to you because I've seen the other people you've talked to and many of them are my role models. I'm glad that maybe I have something that somebody would want to hear, but the people you've had are really impressive. So what are the big lessons that you have learned? Like, I know it's hard to put into a nutshell, but what are the themes that have emerged from these conversations that you've been having? That's a great question. I don't have a pre-prepared answer, but there are definitely a few things that come to my mind when I hear you asking this question. I think the first thing that I learned is that many people in many different areas or many different niches are facing really very similar problems, but they tend to think or frame these problems using different language, but not only different language in terms of different terms for the same meaning, but also slightly different perspective. And I think that's natural in a way because their experiences are different. And when you were talking now about the data and the fact that some people need to engage with the data, very kind of hands on. Other people think deeply about problems and nuances and so on and so on. Their experiences build their perspective. And so I think that's a good thing and that's a challenging thing at the same time. It's a good thing because their perspective allows them to see details that are very difficult to see for someone who doesn't have their experiences and that allows all of us to learn from them. And it's a challenging thing because when we build this channel vision for ourselves, it's very easy for us to start missing the bigger picture. And I'm often encountering people on the theory side and on the applied side. And sometimes I hear from some of these people, not maybe not very often, which I think is a good thing, that they think that the theory is too far away from practice or that practice is just a very special case of a theory that some other people are working on. And sometimes when these people ask me the question, what I think about it, what I think about theory or what I think about this more practical perspective. One thing that comes to my mind is my experience as a music producer. So I used to be a music producer for over a decade. I was lucky enough to work with some really amazing artists from pretty much all over the world. And among these artists, I have people who, sometimes where others, music school, or sometimes they were completely self-taught. And they were people who you would probably describe as a natural talent, someone who comes to the studio and just you sing the melody for them, they play it right away with the expression, you know, when whatever, the most perfect thing you can think of. Some of these people were very, they didn't like theory because they didn't resonate with them or maybe at some point they didn't learn some basics and then it was difficult for them to catch up. And I noticed that for these people who were very, very well kind of embodied musicians, so to say, sometimes it was very difficult to come up with ideas where they got stuck into something. That was very hard for them to go out from this little place where they got stuck. On the other hand, I also worked with people who were very good composers. They were very strong theoretically. And for these people, sometimes it was difficult to get back to the experience of the listener, how they will perceive the music, right? And I think each of these groups has something to learn from the other group because theory gives you an opportunity to think about solutions that are not obvious. It's not something that you can easily come up with using your intuition that is often based on our previous experiences. So I can theoretically build a bridge to something very strange in a way, right? But it makes theoretically sense. And then maybe if we play this progression of chords, that seems very, an obvious like, "Oh, why I would put F# here?" I have F# major when we are in a minor for those people who understand harmony. But then it turns out it makes sense, right? We can put a strong melody on this. And suddenly like, wow, it opens something new, a new universe for us. So I think that's one of the things that maybe some people that are really really focused on practical applications might miss out on when it comes to theory. And in the same sense, fear additions can miss out sometimes on opportunities by not looking carefully at practical problems and the limitations that often come with these problems. So I had a conversation with Mark Van der Lahn recently and he said, "Well, my work is motivated by real world problems, right?" I was starting with very abstract methods. And then I met Jamie Robbins and he told me like, "Well, nobody cares about it because this is not how data looks in practice." And he said that for him, that was a defining moment in his career. So I think that's one of the lessons or one of the inspirations that these conversations gave me to think about the relationship between theory and practice. And I think there is a lot of openness in all of these people I spoke with. I think sometimes maybe we know someone from the internet and they present their opinions in a way that might seem very radical. Or, well, maybe there's no space for debate. We can have an impression. But then often you talk with them and you see how much new ones they really, really understand. - Yeah, that's true. - Yeah. So I think that's one of the greatest learnings for me from these conversations. - That reminded me of something that came up in our previous conversation that I would like to throw out there if that's okay, which is one of the things that's preventing people from, you know, if the goal here is to get theoreticians and applied methodologists talking to each other more, or at least talking the same language, one of the barriers is the availability of data. So, one of the reasons to go work in industry these days, in my opinion, as I'm supervising students or whatever, is that you have access to actual data. And data sets are not made widely available in a particularly useful way, necessarily, certainly not for pharmaceutical studies, certainly not for industry, AB tests, and, you know, everywhere in between. So I don't want to say there's not data out there, but, you know, as an academic, it's something you run into. Like, if you don't have an in-house data generating stream, like they have at these places, and that's a real limitation, and maybe it means that academic methodology just can't go forward in the way that it has in the past, because you can't do good methods if you literally have no access to the types of data that there are. So, yeah, anyway, that's kind of a practical obstacle. But I thought it was worth mentioning, 'cause I know that there are people in your audience that are work on both sides of that divide, and they might have something constructive to say about it. But, you know, it's one thing to tell a student, a PhD student, or a starting data scientist, like, get your hands on some data. But until you have that job, and then when you have that job, you have access to it. It's just a proprietary data is a really interesting thing. To the extent that what you can learn is one of the bottlenecks there is the type of data you have available. It's a really interesting question. In particular, if scientists are not generating the data themselves, and this is a huge problem right now, is that people are trying to do science without generating their own data, and it's because it's too expensive for an individual person. But what a drug company finds expedient and profitable to measure is not necessarily the stuff that a scientist would want to measure to learn the most about this particular thing. And I can only imagine that the same thing holds for a psychologist would want to measure different things for an online experiment. Then the company would necessarily want to measure. So we seem to live in a day and age where big financial interests are dictating the majority of the data that gets collected, which is fine, it's just the way it is in it. But it's interesting to imagine that. I mean, it means that what you have available to generate hypotheses from and to explore is not necessarily what you would measure if you got to call the shots. Sorry, that was a little bit off theme, but I wanted to get that out there because I actually think it's a really important point that-- That's a feature-level selection bias. Yeah, yeah, yeah. I think so, did you just make that phrase up or is that a suspect? No, it's perfect. So feature-level selection bias, I think, is one of the major problems in particular heterogeneous treatment effect estimation. That's not the most optimistic ending we could wish for. Nevertheless, hopefully, it's an inspiring ending for at least some of our listeners. And I hope that we have among them people who, for whom this problem is also close to their heart. Maybe this conversation and this ending will add a little seed for the better future of the causal inference community. Yeah, I mean, I hope so. It's always fun talking to you. Thank you, Richard. I really appreciate it.