What can physics tell us about the brain? - Part 2

Welcome to part two of our episodes with Christopher Lien, assistant professor of physics at Yale University. Now in these two episodes Christopher's taking the idea of statistical mechanics and the ideas of information theory, he's putting them together to try and better understand how our brain works. The really interesting thing about this approach is it's not an approach of trying to understand what individual neurons are doing and trying to build up a picture of what the macro brain is doing. Instead what you do is you're starting at the macro brain and you're trying to use these certain techniques to try and infer what could be happening at a micro level in the brain. This is simplifying complexity. A podcast where we explore the underlying principles of complex systems. Systems that seem to defy our rational view of the world like economies, ecologies, or even you or me. I'm forensic engineer Sean Brady and I'll be your host. Okay so we've got information, we've got uncertainty and we have to find them and we've got entropy, both for information and for statistical mechanics. Yeah and so I mentioned so the key next step is that at the very outset of information theory indeed in the very very first paper where Shannon laid it all out, he called this entropy because he says, oh look this looks like what we talk about in statistical mechanics, I'm going to call it entropy. So there's this surface level connection and forever a confusing similarity in what we call these things and then it was for another nine years until this connection was actually realized to be a bit deeper than this surface level, this formula looks like this formula. So I'm what I'm referring to here is a paper by this guy Edward James and so this is about nine years after Shannon's papers. So now we're talking about I think 1957 and James writes this paper with a really cool title which is information theory and statistical mechanics. It's the title of the paper. You might think it might be relevant for the conversation where it happened and indeed he shows that it's this really beautiful paper where he shows that in fact you can derive the main sort of mathematical tools of statistical mechanics from information theory and the idea here is if and I'm going to try to explain this in a simple term just possible but the idea is if you take a distribution over states let's say and you say well I want to constrain this distribution so that it matches something that I see. So for example when I say constrain you gain some information and then that tells you well I should update my probability over states which means I should constrain my distribution a little bit and then we already saw this with the dice when previously I thought there was a one in six chance and then you tell me oh it's it's actually an even number so this constrains my what I think the distribution over states should be it should be a two or four or six so I update I go from updating my distribution from one in six overall six things to one in three over the only just the even numbers so if I constrain some property of this distribution but otherwise I haven't gained any other information so I should be as uncertain as possible other than the information I got so if you tell me the numbers even I shouldn't say well there's a one half chance that it's or like let's say let's say a two-thirds chance that it's two and then a one in six chances for it and one in six chance that it's six it should have an add-up to one that isn't quite right like there's no reason I should favor two over four and six because you don't have the information yeah no one has provided information to justify that bias so really what I should do is I should say well I should take in this information I should constrain my distribution to two four and six and then otherwise I should be as random as possible I should just say well one third chance up to one third chance up four one third chance of six so what Jane shows is that if you take this process of taking in information constraining your distribution and otherwise being as random as possible what you get out is a new distribution that looks mathematically exactly the same as the types of distributions we see that are at the core of statistical mechanics which are known as Boltzmann distributions so Boltzmann was this sort of fathers of statistical mechanics back in the 1800s and he so he gets a distribution named after him and so the distribution over states in statistical mechanics is given by this distribution that just depends on the energy of the different states the energy you can think of as just telling you which states are more or less likely so high energy states are less likely low energy states are more likely and then the probability of those states is just some function of those energies so it turns out that if from just a purely information theoretic perspective if you tell me so if I suppose I don't know I have no idea what state the system is and so I have it what you would say is a uniform distribution over states it's just like the dice we have a one in six chance over anything it's a uniform distribution but now you tell me we're talking about now a statistical mechanical system like a physical system like these molecules bouncing around if you tell me what the average energy is and that's the only thing you tell me this provides some information to me it doesn't tell now I shouldn't just be uniform over all the states like I should make sure that whatever distribution I have in my head when I add it all up and take the average of the of the energies it should equal what you just told me because you've just given me some information so I should make sure I take that into account so the average energy is some number so when I take that into account but I'm otherwise as random as possible just like Jane said we should be I'm as uncertain as possible but I fix this average energy it turns out you get exactly the Boltzmann distribution in disco mechanics so the Boltzmann distribution is nothing but the most random distribution you could have given an average energy that's the statement there's a lot I know there's a lot I know I just said a lot of things so when we say the average energy we're talking about a macro state now yeah and then a year saying there's multiple different micro states that could produce that macro state so you you got to represent the probability essentially of what they would look like and you do that and when you do that you end up with this Boltzmann distribution and what that is is that saying the answer could be anything but because I now know the rant that the average energy I have information at a macro state and I can use that to inform me of the probabilities of the micro state but I'm going to go no further than the information I currently possess exactly that's exactly right yeah okay yeah so you got it as random as you can as random as you can but consistent with the information you just gave me so it's just like you told me it's an even number on the dice I should make sure I only put probabilities on the even numbers because I now I know it's not an odd number so I should throw those away but other than that I should still be as random as possible so what's the significance of that thing why is it significant that we get a Boltzmann curve yeah so the significance is it's kind of twofold so from a more philosophical perspective what it basically means is that in some sense we can derive statistical mechanics from information theory which you might think means that information theory is somehow perhaps and this is a bit controversial because I don't know if every physicist would agree with this that information theory in some sense you can think of as a bit more fundamental than statistical mechanics because you can take information theory and you can apply it in a specific case and you get statistical mechanics so from a fundamental perspective it gives us this statistical mechanics is like a special application of information theory when viewed in this way so that's the first reason that this is perhaps important from a practical perspective from a person doing day-to-day things trying to calculate things what it means is that when we build models using information theory when we try to think about information and for example biological systems or the brain then oftentimes we can lean on this connection to statistical mechanics because it's such an old field it goes all the way back to the 1800s because this wealth of knowledge about how to deal with these kinds of systems from a practical perspective how should we build models of these systems what are some mathematical tricks for computing things that we might be interested in and so sometimes when you frame something about a system that's not a statistical mechanical system it's like the brain or something if we frame things in a certain way using information theory often the models that pop out look like models in statistical mechanics that then we can use the math of statistical mechanics to make progress so that's from a practical perspective the big helpful deal here so that's mad and I mean it's complexity science sort of at its best and you know you're going from one field to the so we have a field of statistical mechanics which start and develop because we want to build steam engines and build more efficiency dimensions and then someone comes up which Shannon Cloud Shannon comes up with information theory which looks like a fundamental part that actually helps us explain statistical mechanics but then we've got all this you know we've done all the maths we've got everything available and then you're saying oh but the brain is going to calculate information and if I apply information to it actually starts to look like statistical mechanics but now I don't have to work out the answers from scratch here I can just go back and use the old techniques that are tried and tested so you're essentially applying steam engine physics yeah that's very crude but you know what I mean to the brain yeah which is exactly actually the motivation we kind of started with which is we had this kind of intuition that perhaps statistical mechanics is really useful for thinking about how you have these simple rules at small scales and then at large scales you have these emergent relationships between temperature and pressure and volume and we hoped that perhaps you might be able to use similar ideas to study the brain and start with perhaps the micro states that a brain could be in and could we derive larger scale predictions for relationships between macroscopic quantities. And indeed, when we phrase things in a certain way using information theory, we can in fact make that analogy hardcore. We can make it mathematically precise. And I should be specific that you have to frame specific questions in a very specific way using information theory. But then when you frame things properly, and you can't answer every question this way, there's a set of questions that if you frame them properly using information theory, things are mathematically equivalent to the models you build or mathematically equivalent to models in statistical mechanics. And so this analogy, in some ways, is no longer an analogy. It's a mathematical equivalence. Well, it is still an analogy. The math is representing two different things. It's a tool that we know works. Exactly. Yeah, we spent a long time getting a PhD in physics. We might as well put it to use to study statistical beginning and a statistical mechanics was not easy. So I might as well use it for something. For something. So go to the brain now. What specifically have you applied this to? So you can already perhaps guess the types of questions where this might be useful. So just to give a quick recap of the connection we made between information theory and statistical mechanics, we said that if we have a bunch of states of the system and we learn one thing, we're provided with one bit of information, which is the average energy. Now I should constrain that in my model, but be maximally random otherwise, be maximally uncertain otherwise. And now we have a Boltzmann distribution like we have from statistical mechanics. So you could ask, well, what if in the brain, there isn't the same notion of energy as you have in statistical mechanics? I mean, there are notions of energy. The brain does burn energy. But it's different than what we think about in equilibrium statistical physics. And we could get into that, but I won't go down that path right now. But you can immediately see how you could constrain. So anytime we receive any amount of information about the brain, so suppose we do an experiment, you could measure, for example, how many neurons on average are active at the same time. So it's a number. So if we look at 1,000 neurons, we're recording from 1,000 neurons well in a mouse while it's doing some task. And you could tell me a number. It's on average. This is how many neurons are firing at a given moment in time. And so you could say that this is the overall average firing rate, basically, of the neurons. That what a neuroscientist might say. That's your macro state of the brain. Yeah. And that defines your macro state. There's many different ways that the micro states could be with different neurons firing that give you the same number of total neurons firing. So there's still the same idea where there's different micro states that give you the same macro state, which is defined by this is how many neurons are firing on average. And what you can do is you can ask, well, given just this bit of knowledge that we got from an experiment, what model should we build of the brain? Or we should build a model where we make sure we get that number right because we know it now. But otherwise, we should be maximally random because we don't have any other pieces of information. We want to make sure, whereas random is possible, while constraining the thing we do know. And if you turn through the math, what pops out the other end is precisely, again, a Boltzmann distribution. But here, instead of constraining the average energy, we've just constrained the average firing rate. So now you have a model where the average firing rate plays exactly the same role as the average energy does in statistical mechanics. And indeed, you can follow this logic. And you can constrain anything you want. You could measure anything you want in the brain. You can strain it in your model. And as long as we're trying to be as random as possible, other than things we measured, what you get is something that looks like a model from statistical mechanics. This is how we could apply these ideas, how close the loop to thinking about models in the brain. Then you might ask, what does this tell us about the brain? But this is mathematically-- Exactly what it's going to ask you. This is mathematically how you get from brain studying a question, using information theory, and then out the other end, it looks like something we know from statistical mechanics. And so then, what does this tell us about the brain? So doing this process, building these kinds of models, it won't tell you everything you want to know about the brain. So I should make sure to be clear that this is not a magic bullet to understand everything about the brain. There are many, many questions that should not be answered this way. But there's a set of questions where this is the right thing to be doing. And so you get asked, well, what are the kinds of questions for which this is the right kind of answer to be looking for? And the kind of questions you might want to answer are, well, if I'm told some information about the firing of these neurons, how do I know what other bits of information sort of emerge from the bits we were just told about? So for example, if you tell me the average firing rate of the neurons, and then I constrain that in my model, but I'm maximally random otherwise. So I build this model, and I have all this nice machinery from statistical mechanics, and we can compute anything we want. Now we could ask, does this model, which basically only has this one bit of information, has this just the average firing rate, but is explicitly as random as possible otherwise? Can we predict other features that we see in the data? So now you could ask, well, does this predict the correlations between neurons, which is something that we don't know a priori? But perhaps if the correlations emerge from the thing that we just measured, I'm going to define an emergent a second. So if you tell me something, and I constrain it in my model, but otherwise I'm maximally random, and then I go and I say, well, now this model actually predicts these other things that I didn't know. Then in some sense, those things can be thought of as emerging from the bit of information that we already constrained. And this is the key step. And so perhaps it makes a bit of sense to go back to the dice example. So for example, I'm making this example up on the fly. So I'm going to try to do it justice without twisting the analogy too much. So in the dice analogy, at the outset, we had no sense of what number it was going to be. You give me one nugget of information, which is an even number. So now we have a model that constrain, as we talked about, but is maximally random other than that little bit of information you told me. So it's 1/3, 2/3, 4/1/3, 6. Now you can ask, well, does this allow me to predict anything else, something else about the dice? Like does it allow me to predict that the number is-- how often I'm going to get it to, for example? And the answer isn't quite yes. So the analogy here isn't great because the answer is not quite yes, but it does give me a better prediction of whether or not it's going to be 2. Because before I gave to a 1/6 chance, and now I give to a 1/3 chance. So you might ask-- and so this gives me a better prediction of whether or not what I'm about to see as a 2. In this way, you can start to see how constraining one thing might-- if that gives you information about another thing, then there's some relationship between the bit of information you gave me and other things we might go in and try to predict. There's another way of thinking about it that if we take a model of a brain, and you apply this average firing rate. But let's say the model you had before was firing all over the place. When you apply that average firing rate, you're obviously reducing down the firing, you're curtailing it. So that in some way has to curtail the underlying connectivity, doesn't it? So if we say look at the average firing rate is lower here than, you see, in this other model over here, or this other brain over here, presumably that means there must be less connectivity in whatever way-- however we define connectivity, so just leave it loose as hell. Because if there was higher connectivity, or whatever, have an average higher firing rate, or something else driving that thing. So is that what we're getting at? We're saying that it's telling us something fundamental about how the structure or the behavior of this system works. Yeah, so it's hard to make inferences directly about the structure of the actual connections between neurons, which is what we were talking about on the last podcast. Precisely for a reason that we talked about on the last podcast, which is that in very few data sets, we actually have the structure of the connections in experiments at the same time as the activity. So it's hard to actually make inferences between the two directly in general, some of those special cases where there's really cool experiments where sometimes you can. But in general, what we're saying is we're just thinking about the activity. We're trying to think about, well, what are the probabilities of different patterns of activity we see in the brain? And when I say a pattern of activity, I mean, well, which neurons are firing or not at a given moment in time? And so if you tell me, for example, the average firing rate, then this gives me information about what the distribution over these patterns should be. This is actually quite important for neural functions. So the patterns of activity are what, for example, in the retina, they encode what you're seeing in the visual world. And then deeper in the brain, they compress this information into sort of representations that we like, oh, I see a book over here. And I see a coffee mug over here. And then we use these representations to make movements and plan and make decisions. So these patterns of activity are, in some sense, are very important for understanding what the brain is actually doing from an activity perspective. So what we're saying is that you could ask yourself, well, what are the key features about these patterns of activity, which, once I can strain that feature, tells me as much information as possible about these patterns of activity. That's sort of what we're trying to get at. So the bits of things we could measure and experiments about these populations, like the average firing rate. The things that provide more information are things that tell us more about what these patterns of activity are going to be. So just like what we talked about before, whereas if you tell me exactly what number of the dice is going to be, that gives me more information. That tells me more than just telling me that, oh, it's even an even number. So equivalently, what are the things we could measure about the activity in the brain that really allow me to predict the patterns of activity we're going to see? These are the. of things that provide us information in a very specific information theoretic sense about the neural activity. - And in fact, hopefully they help with tell us something about how the brain works. - And have you got any results? Yes, that are interesting? - Yeah, so a classic result along these lines, there's many versions of this type, this flavor of question. So, because there's for any type of thing you can strain in your model, anything we measure in the data that you can strain in your model, you could then ask, well, what can we predict from that using these types of models? But there's a wealth of different things we could constrain. So the simplest thing you might constrain actually is the firing rate of each individual neuron. So say we're looking at a thousand neurons, you tell me that in any given moment in time, what's the probability that each neuron is firing, which is equivalent to the firing rate of the neuron? And you could say, well, what if we just constrain those numbers, those measurements in our model and ask, is this an accurate model only knowing this and being maximally random otherwise, is this a good model of these neurons? And these models allow us to answer this question. And so the answer to this is no. That is not a good model. And we already learn about the system, but by learning that the answer is no, because what that means is that there's other features in the neural activity, other things we could have measured that give us more information that are really, really important for what these neurons are doing, that we've missed, we haven't included them yet. So you've got a model of a brain, you've got a system which is a collection of neurons and they're connected in some way, stop me from running here. And you measure the firing rate of each one of the neurons in a real brain presumably. So that's the probability to each neuron. And that's obviously statistical. And then you run that model. And what are you seeing that's telling you, well, that's giving crazy behavior. It's not giving behavior that makes any sense. Exactly. That's exactly the right question. So where is this model failing? Is exactly the right question. So it actually goes right back to our two coins. So if you tell me coin one is going to be head to some probability and coin two is going to be tails with some probability. And then I build the maximally random model I could with that information. The maximally random thing those two coins could be is independent. And this is something that I didn't tell you before, but this is a key fact about probabilities and information. It's that the most random two things could be is independent. So if there's any correlation between those two coins, that actually reduces how independent they are. I mean, reduces our uncertainty. So correlations reduce our uncertainty. They provide information in some sense. So if the only numbers we include in our model, the only things we measure are the firing rates of these different neurons. That's exactly the same as the probability of the heads versus the tails. And the most random model we could build is just a bunch of independent neurons firing exactly the same way. So the most random model will say, well, these neurons should just fire with some probability that you told me. But otherwise, they should be totally independent. Whereas when we look in the real brain, these neurons are very, very strongly correlated. Or some of them are very, very strongly correlated. Because like you said, many of them presumably, they might be connected very strongly. They might be receiving some common input from a visual stimulus on the retina. Then they're going to be very correlated. So across all different types of neurons we look at, there's very often very strong correlations. This tells us that what we've missed in our model, so it's not just enough to know the firing rates of these neurons. We've missed a key feature, which is the correlations between the neurons that we are doing a horrible job of predicting, which means there's information we need to include, which is encoded in these correlations between the neurons. So by ignoring it, your model doesn't work at all. Exactly. And you now know that obviously we can't ignore that this is terribly important. Exactly. So then do you start looking at things like the probabilities of this neuron firing when X other neurons is firing? Can you start to do that sort of stuff? Yeah, exactly. And so exactly like what you're saying, given that neuron one fired, what's the probability neuron one hundred seventeen is also fired at the same time. This is precisely what I mean when I say correlation. So those are one and the same. So now you could ask, well, what if I include the correlation between neuron one and neuron one hundred seventeen? Suppose I include that in our model. How much information does that provide? And this is a very well defined question. We can write down a number, which is how much information that correlation tells us about the activity of the whole population. And so then you can start to ask, well, what are the really important correlations between these neurons? Which are the correlations that provide us the most information and tell us as much as possible about the whole pattern of activity as a whole, the population as a whole? So you can start to ask, which are the really, really important correlations? And that's exactly what these models allow us to do. What I'm describing now at this point is we've brought you right up to the forefront of current research. So we've taken this long winding road, but this question I just described, which correlations provide us the most information is literally a paper that is not even published yet. So these are things that we're actively, in fact, I'm supposed to be writing this paper right now. (laughing) Hey, you're doing what all good writers do. You're talking about writing the paper instead of writing the paper. When we get off the call, that's what I'm about to go do. Exactly. Anyone out there don't steal, don't go steal this idea. (laughing) If we go back to our last episode, and we talk about, you know, applying network theory, so we know we've got this incredibly rich area of network theory that we understand how systems would interact in components can arrange themselves, not all those sorts of things. I mean, that's a very bottom up sort of way of trying to understand the system and how the system puts together. And I mean, this is really coming at it from the other end of the scale, isn't it? It's almost measuring the effect and saying, you know, we've got really good tools at measuring the effect, like pressure and temperature and a gas. And we've got really good relationships with tying that back to what's producing that effect, what interactions are producing that effect. And then we're gonna take that to the brain, look at the scans that we see coming out of, look at the activity we see, come out, and then tying that the whole way back. I mean, that's incredible. That's actually exactly right. That's actually a really good insight. So it is true that what we were talking about in the last episode, which is thinking about the actual connections between the neurons. And we even talked about digital twins. And if I know all the connections between the neurons, and I think I have an idea of how neurons work, well, I could just simulate the whole network and see, well, and then this is a very bottom up approach. It's sort of, we're starting with the very ground truth. We know what the wires are. We think we know what the biology is. So can we simulate starting from the bottom all the way up? Can we simulate a realistic looking brain? And this is exactly the opposite direction. We're saying, well, let's just measure a brain and try to infer what the important underlying bits are. What are the important correlations between the neurons? Starting from the top, could we run this whole process backwards and infer what the really important things are at small scales? And in fact, the problem that I just described is there in statistical physics, the lingo, that it's often called the inverse problem. So you're actually inverting the whole problem. You're turning it all upside down and trying to start from the top and infer the underlying microscopic details. So it's exactly roughly the opposite problem from what we talked about last time. - That's brilliant. What does real success look like for you in applying this? What will the paper say when you nail it? - Yeah, that's a great question. There's kind of two classes of questions. We can talk about the bottom up or we can talk about the top down. I mean, ideally we can meet in the middle and we have a full understanding of everything. I guess that would be the holy grail. But in the shorter term, well, I guess I could sort of take a step back and try to be more speculative and grandiose. So what I think would be the coolest thing to do or to understand is if we really had a way of looking at neural activity and figuring out being able to, like we said, run this whole process, the bottom up process backwards. So do this whole top down thing and could we have a really good idea of what's going on at a small scale just for measuring the activity of the neurons at a large scale? And does that line up with when we know what's going on at this small scale and we think we have these biological knowledge and we run the forward process, which is the bottom up process. And we try to simulate the system. Do these two pictures agree with one another? 'Cause if they don't, then something's wrong. Something in the top down picture or the bottom up picture is wrong if they don't agree with one another. And so I think these are the really big questions. And in fact, this is exactly sort of what we started off talking about with statistical mechanics. The whole goal here is to bridge two scales. It's to bridge the scale of the individual neurons with the scale of the whole brain and these sort of macroscopic properties. And we talked on the last podcast about how, when we talk about humans on Earth, thinking about individual humans is an entirely different academic discipline known as psychology or cognitive science. And thinking about all seven billion people on Earth is the realm of political science or sociology. And you might think about trying to bridge those two. In neuroscience, we have an even bigger gulf in scales between individual cells and then all of these cells acting together with the hundred billion neurons in our brain. And can we bridge these two scales? Do we know enough at a small scale to predict the large scale? If all you told me was the large scale, could I have inferred what the small scale details are? So that, I think, as a field is perhaps the biggest question in neuroscience. Can we bridge these two scales? This won't be one paper, though. So I think I took your question and ran with it because this will not be solved in one paper, but that's the goal. So hopefully if you can take, when you chip away, if you can take a meaningful chunk out of that question, that would be a pretty cool paper. Christopher, thank you very much for being on the show. Thank you for having me. There's been a lot of fun. [MUSIC PLAYING] Thanks for listening to Simplifying Complexity, where we look at the key concepts of complexity science with expert minds from across the world. Concepts like emergence, self-organization, adaptation, networks, scaling, tipping points, and much more. This podcast was produced by Brady Heywood and Wevelyn Creative. To make sure you don't miss an episode, be sure to subscribe to or follow the show in your podcast app. I'm Sean Brady, and I'll see you in our next episode. [MUSIC PLAYING]