Movies, neurons, and AI

Welcome to Science Sessions, the podcast of the proceedings of the National Academy of Sciences, where we connect you with Academy members, researchers, and policymakers. Join us as we explore the stories behind the science. I'm Paul Gabrielson. Understanding what's going on in moving images or movies seems effortless to us, but is challenging for AI systems. Processing movies requires integrating changing information over time, a process that is computationally intense for AI. In a recent PNAS study, Hollis Klein and Masaki Hiromoto of the Scripps Research Institute studied how the neurons of Xenopus frog tadpoles process moving images. Then they used what they learned to develop brain-based AI image encoding algorithms. The algorithms, the authors say, reduce the data and processing time needed to recognize moving images. Hollis, what's the background of this study? There have been many studies which have investigated how the visual system detects information in our world. And many of these studies have focused on relatively simple kinds of input like a moving stimulus across the eye or the visual field. This information has been extremely valuable and shown fundamental rules by which sensory information is processed in the brain. This information about how the visual system works has been extremely instrumental in developing machine learning and artificial intelligence, for instance, for facial recognition or using AI to detect and compare single images. What are the current limitations in how AI recognizes movies? AI, as I just mentioned, has been extremely advanced with respect to detecting individual visual scenes with no motion. But when you add a time domain to this kind of information, it's extremely challenging for machine learning algorithms to process this and to compare it across different kinds of samples, even with extensive training. The real motivation in our study was to understand more about fundamental mechanisms by which the visual system detects and encodes information. We didn't think about the application to artificial intelligence until we had actually made the first discoveries with respect to visual system function and the biology, the neuroscience, behind visual system information processing. Then we realized that it may be applicable to machine learning algorithms. Why did you study the visual systems of Xenopus tadpoles? Xenopus tadpoles have a relatively simple visual system compared to us or other mammalian species. And yet, because of evolution, their visual system works extremely well to detect and process a variety of different kinds of stimuli. So they're a relatively simple experimental system for us to use. And yet, they provided a lot of depth of information to us. How did your experiment work? So there's a tadpole. It is sitting on a platform with a specialized electrode entering into the brain. And next to the animal, the eyes are on the side of the head, next to the animal where it can see is a like a movie projector or an iPad basically showing this variety of essentially static light images from which we use the reverse correlation to determine the optimal stimulus. And we record for like up to 16 hours from these neurons. So we didn't start out by showing them movies. We started out by showing them what's called a sparse noise stimulus. So you could think of that as just a bunch of scattered bits of light across space. And we used electrophysiological recordings from single neurons in the visual center of the tadpole brain called the optic tectum. And we recorded the responses to these sort of scattered bits of light and dark over many different hours. We then took the recordings and analyzed to determine what was the optimal stimulus that made that neuron fire action potentials. The neurons have a preferred stimulus. And what we observed was that the neurons have a very long temporal domain for detecting visual information. Well, like six or seven hundred milliseconds. So we call that long. Because most of the stimuli are presented for maybe 10 or 20 milliseconds. The other very interesting thing that we learned from these experiments is that the preferred visual stimulus was bookended by a light on stimulus and then it was terminated by a light off stimulus. But what was interesting was in between that beginning and end, the preferred stimulus was quite variable from neuron to neuron. And so this basically indicated that there were fundamental rules in terms of what was information that was taken into these neurons. And yet there was a lot of flexibility in terms of what we call the optimal stimulus that would be detected by these neurons. What did you see in their neural responses? The neurons responded to quite a variety of different stimuli. One thing that was kind of interesting was that about 30% of the cells responded to a rotating stimulus. So we tried to imagine what this might mean for the life of a tadpole. And if it's swimming along, there might be something in the pond where the animal lives that would be rotating in an eddy of water or something like this. And so we think that these different stimuli are representative of the environment, the salient features of the environment that the animal lives in. How did these neural responses change with training? Much of the work that we've done in my lab has been an effort to understand how the visual system changes or is plastic over time. And a classical way of doing that is by providing a particular type of visual experience and then testing whether or not that has an influence on how visual responses are detected. So we recorded the favorite responses of particular neuron, and it would, let's say it would have a beginning and end and it would last about 300 milliseconds. We either shortened that movie to maybe 200 milliseconds, or we extended that movie to maybe 500 milliseconds, and we trained the animal repeatedly with either the shorter or the longer movie version of that same information. And then we tested whether or not those same neurons that had the 300 millisecond preferred movie could detect the shorter one or the longer one. And it could do both of that. So the point is that these neurons have a preferred visual stimulus, but they still respond to changes in their environment and can demonstrate plasticity and response that changes in the environment. How did this lead to a machine learning network for movie recognition? We were inspired by past history of critical events and AI development. There are two cases in which fundamental features of the nervous system development and function have been applied to AI to help make critical advances in machine learning. And one was the Gabor filter, which is a way of describing changes in frequency of light from dark to light to dark. And another is the convoluted neural network, which was developed by Fukushima, based on his knowledge of the topographic map formation, actually, within the frog visual system. When we realized that these frog neurons could encode complex visual information, we thought, maybe we can take this fundamental encoding principle and use it to build essentially a new version of the convoluted neural network to apply to machine learning. And so what we did here was make a matrix of each of these different visual neuron response properties and put that into an algorithm to detect motion stimuli. What can this machine learning network do that others couldn't before? It's capable of detecting a wide variety of motion stimuli of movies. Brains of all animals are highly efficient at detecting and coding and storing information, much more efficient than any of the computers that have been built so far. AI is extremely computationally expensive, and this is really a limiting factor now. By applying these brain principles and with the inherent efficiency that the brain has, this could help make AI or machine learning more efficient. What are the caveats or limitations of the study? Every advancement in science has to be reproduced. and corroborated by independent investigators. And I look forward to additional scientists taking the principles that we've had and testing whether or not they can replicate them. We've already begun to think of how to improve the algorithms that we have generated. And so we're working on that in parallel. - Thanks for tuning into science sessions. You can subscribe to science sessions on iTunes, Spotify, or wherever you get your podcasts. If you like this episode, please consider leaving a review and helping us spread the word.