Can AI help us speak to animals? Karen Bakker interview

the ability to record animal sounds and use this to try to communicate with them seems very tantalizing, you know, a lot of people have a deep interest in nature and a sort of deep yearning to maybe understand whether there's species might be saying, but there are great risks that I think we should be wary of. That's Karen Backer, the eminent Canadian academic, an author of a book called The Sounds of Life about how digital technology is helping us to understand the non-human world. Karen was a leading voice in the field known as bioacoustics. In fact, her book was what inspired us in the first place to make this later series about using AI to one day speak with animals. But while we were putting this series together, we learnt the tragic news that Karen had passed away. We used extracts from her interview in the previous two episodes, but the whole conversation was so interesting, we wanted to share it with you. I'm John Thondhill, the financial times innovation editor, and this is tectonic. The interview you're going to hear with Karen wasn't my first. I spoke with her on stage at the FT Weekend Festival in Washington, DC in May this year, where she mesmerized the audience with her astonishing expertise and rare eloquence and some spine tingling recordings of bats and whales. As her colleagues at the University of British Columbia have said, "Karen's immense energy, passionate intellect will be dearly missed." This interview, broadcast with the blessing of Karen's family, is our own tribute to a remarkable woman. In your book you start off with this kind of wonderful image of this kind of planetary conversation that has been going on all around us for a very, very long time, which we haven't been able to listen into. Could you just tell us a bit about that planetary conversation and how we are now able to listen into this thanks to machine tools? So, if you pick up your smartphone, we think of this as a device for communicating with other humans, and it is. But the same technology that's in your smartphone, including microphones, accelerometers, is now being used to track and monitor the lives of other creatures on this planet. So, if you think of the Internet of Things, being ubiquitous in our lives, imagine the Internet of Earthlings, in which we extend monitoring and sensing capacity to all living things on the planet. What this is doing is revealing a remarkable set of capacities amongst other species, and one of those capacities is complex communication, much of which occurs at frequencies which we are not able to hear. So, much communication in nature occurs in the high ultra sound above our hearing range, or in the deep infrasound below our hearing range. At the high frequencies, Moths, mice, beetles, rats, Tars ears, these little primate cousins of ours, at the low frequency, whales, elephants, tigers, even peacocks are all communicating in sound, which we simply cannot hear, and because humans tend to believe that what we cannot perceive does not exist. These sounds have largely passed us by until the advent of digital bioacoustics, which uses pretty much the same tech in your smartphone to listen to these hidden sounds of nature, and we're uncovering some marvelous things. And you have this kind of wonderful analogy in your book that it's as though our radio frequency has only just been set on one dial, and now we can move along the dial, is that right? Could you just explain that to our listeners? Yes, so Leroy Littlebear, who's a Blackfoot philosopher, has this wonderful insight that the human mind is fixed and narrow as if we were just listening to one station on the radio dial in a tentative or unaware of all of the other species communicating in a range of other frequencies. And if we could just turn the dial and tune into those other stations, we'd be able to access remarkable types of communication across the acoustic spectrum and across the tree of life. And that is what digital technology allows us to do. This kind of revolution in bioacoustics has been kind of fueled by two changes. One is the kind of ubiquity of microphones, which is collecting data on the scale that we've never done before. And two is the application of machine learning systems to that data. So could you just walk us through both of those first of all, how do we harness the data in the first place? And then secondly, how will we get in clever at analyzing that data? Yes, so the use of digital bioacoustics simply accelerates something humans have long done listening to nature is, of course, an ancient art doesn't require digital tech. But what digital listening devices do is they enable the automated low cost ubiquitous and continuous recording of nature sounds even in remote environments. And that creates an abundance of digital acoustic data. So scientists are installing large arrays in West African forests to listen for forest elephants or deep in the ocean to listen for whales. So with these very large data sets comes a new challenge, a wonderful new challenge, there's simply too much data to analyze manually. But with the aid of machine learning, so a specific form of artificial intelligence, we can automate the analysis of this data and search for patterns, acoustic patterns, communicative patterns, and build dictionaries. So scientists are building a dictionary of sperm whale bioacoustics. They sound a lot like Morse code. You can use the same principle to study lots of other species. So there's now an elephant dictionary with thousands of elephant sounds. And once you begin to parse those patterns and associate them with behavior, you start building up an understanding of what the communicative regimes of other creatures look like. So for example, that elephants have a specific signal for honeybee. Elephants will make very specific sounds when in estrus, when in a bonding ceremony at the birth of a new baby. So this is pulling back the curtain, if you like, on these complex social lives of these animals, as we begin to understand the patterns in their communication, many of which escaped us before the advent of these automated tools that essentially allow us to see. The landscape of communication of another creature at scale. Now we've been arguing for a very long time about the nature of language, what it is and how we communicate and so on. And I thought it was very interesting the phrase you used just now that you were talking about the communicative regimes of different species. Are they using language? So the question of the definition of language is authority one and philosophers, linguists, psychologists will all weigh in. Many will argue from those disciplines that there is something unique about human language, be it the symbolic content, the ability to abstract concepts, you know, sort of combinatorial capacities. Scientists who study bioacoustics don't actually talk about this definition of language very much. And that is in distinction to where the debate was 20 or 30 years ago when we sought in a very anthropocentric way to assess the communicative abilities of other species by seeing if they could learn human language or if they had some kind of language that was like hours. This new generation of scientists is interested in the communication of other species on their own terms from their own worldview or umvelled the sort of embodied experience of being in the world. So one could argue that the question of language is actually slightly irrelevant. That is, we can deeply appreciate and learn much about the lives of creatures like bats or elephants by immersing ourselves in their communicative worlds. Now, it turns out that as we do so, we're realizing that their communication is far more complex than we realized and the ability to sense complex ecological information conveyed through sound extends much more widely across the tree of life to insects and even plants than we realized. So we may one day arrive at a moment where we redefine the concept of language in less human centered terms. But in a way we've transcended the debate, we can appreciate the marvelous complex communication of other species without really needing to if you like hit the wall of whether or not we would include them in the human perspective of language. Now, one thing I'll say just about that, there's a wonderful quote from Victor Stein who says, you know, if a lion could speak, we could not understand them. And of course, Nagle argued that we, even if bats could communicate, we'd never understand them because we do not live like a bat. We don't fly, roost upside down. We're not embodied like a bat. But they wrote those arguments before the advent of digital technology. You and I could never echolocate like a bat, trumpet like an elephant buzz like a bee, but our computers can and they offer us these translation devices that were not available when many of these arguments about the uniqueness of human language were formulated. So I argue in the book that those debates are being destabilized by these new digital technologies. And as you're rightly warning against going at dangers of anthropomorphizing what we're doing and thinking it's all about us. But are there examples that we know of of interspecies communication already happening between plants and animals or between animals themselves? Yes, there are examples of interspecies communication. So in nature silence is an illusion because sound is everywhere. If you could hear all these frequencies, you would hear the sounds being generated by plants, which many insects can hear, but we cannot. You would hear the sounds of our planet, which makes infrastructure sound thanks to volcanoes and earthquakes. So there's all sorts of low sounds that you and I cannot hear, but some animals can. So there's just like we cannot see in the ultraviolet. We cannot hear in a lot of these frequencies. So there's an enormous amount of acoustic information that's circulating and some of this allows communication within species, some of it between. So one example would be the acoustic tuning communication between predators and prey bats will use sound to echolocate when they hunt moths, but moths can also jam bat sonar. There's acoustic tuning between pollinators and pollinated plants. There's also really interesting evidence of acoustic sensing between plants and insects. So Heidi at the University of Toledo has done wonderful experiments, very ingenious, showing the degree to which plants are sensitive to the sounds of insects and they can discern with a high degree of acuity, the sounds of non-threatening environmental sounds like rainfall and the sounds of threatening environmental sounds like the sound of an insect. That is a predator of their plant chewing on plant leaves and only released defensive biochemicals when needed. So all of these are examples of the transmission of complex ecological information between species that we're only just waking up to. One of the things that comes out in your book as well is that our ancestors were far more attuned to kind of sound than we are. And that's also true of a lot of indigenous people as well. I mean, we have moved as you described from relying on oral communication to visual communication by reading and so on. But you talk about a resurgence of an oral culture. Could you just explain that to us? So Western science and Western culture tends to privilege sight over hearing and the visual over the acoustic to some extent that can be attributed to a long evolutionary process of us as terrestrial creatures, but more recently to a cultural process whereby the advent of the printing press and the channeling of culture into written texts also leads us to privilege. Textuality over orality. So with that, what may be latent capacities to listen, have been attenuated, we can really get a sense of this when indigenous knowledge reveals that human listening capacities can be cultivated to be far more sensitive than we experience in Western society. So in the book, I talk about some wonderful examples in Brazil where specific tribes cultivate the art of listening from an early age and their sense of hearing allows individuals to hear the sounds of let's say fish in an environment where the sounds would be completely inaudible to Western anthropologists. Similarly, there is evidence of humans being able to echo range, particularly humans born blind, contemporary humans in Western society do not cultivate these capacities, which may be dormant or it may have simply vanished. If we look back further in evolutionary time and along the tree of life, we see that some of our primate cousins, like Tarsures, these are these tiny creatures with pointy ears and these very cute large round eyes, they can listen and communicate using ultrasound so it may be that our human ancestors once had this capacity, which we have since lost. Digital listening devices function like a hearing aid, like a prosthetic that extends our hearing capacity beyond the sensory limitations of our bodies. And at the same time, as I note in the book, digital listening is not the only way to access these sounds as indigenous communities show us, there are other forms of deep listening that are also very powerful forms of revealing the hidden sounds of nature. When I interviewed you on the phone a while back, you came up with this wonderful phrase that sonics is the new optics, but in a double sense, I mean, one is that you're in a way saying kind of sonics is the old optics. And do you think we're moving into a world where we combine both sonics and optics in a way that we haven't been able to before. Yeah, let me explain that a little bit. So the importance of digital bioacoustics can only be understood in a historical context. And let me give you a historical analogy. So about 400 years ago, the inventors of the microscope were astonished to discover the microbial world. Van Leuwenhoek originally kept his discovery secret for fear of being thought crazy and for a time he was. So at that time, the early inventors of the microscope had no idea their invention would eventually lead to the discovery of DNA and the ability to manipulate decode and manipulate the code of life at this around the same time, the inventors of the telescope were gazing up at the stars, not realizing that their device would one day allow humanity to look back in time. Nearly to the origins of the universe optics was profoundly important for the scientific revolution, but also for de centering the concept of the human within the solar system and within the cosmos with a number of course cultural and religious implications. In the sounds of life, I argue that sonics is the equivalent of optics in so far as it de centers humanity within the tree of life. We're only at the beginning of exploring this vast new scientific terrain, but it's already revealing some hugely surprising insights about the large number of species that communicate that we thought were mute and deaf. It's also revealing the existence of complex ecological information and much more complex social lives and other creatures than we previously realized. And as we discussed, it's also challenging notions of human uniqueness about language. All of this de centers humanity from our supposedly privileged position within the tree of life and the sonics techniques can be applied to many other fields, so your listeners might be aware of the growth of processes of sonification of data, the very beautiful work that's being done to sonify data to listen to the universe in physics. So there are many, many applications of sonics that go beyond biology or ecology and apply across the sciences, given the sort of the primordial importance of not only acoustics, but chromology, microtomology, biotromology, vibrations. So this is leading to, I think, a very exciting new set of scientific advances that may turn out to be as important as optics was several centuries ago. I'd like to move on to the kind of AI side of this. Could you explain to us how are you using machine learning tools to deepen our understanding of what's going on in bio acoustics. So imagine using a set of digital recorders out in the ocean to record whale sounds. You're probably familiar with humpback whale sounds, these long, beautiful songs that they sing, but many other whales make sounds. For example, sperm whale bioacoustic sounds a lot more like Morse code. Now if you were to listen in on these recordings, it would be quite confusing. In part because the whales are all singing or clicking at once, how do you know which whale is making which sound, how do you know what they're doing when they make the sounds is all happening deep underwater. But with the advent of the digital listening tools combined with machine learning or artificial intelligence to analyze the data, we can do a few things. First of all, the machines can identify which animal is making which sounds. So they each have a voice print much like you do algorithms can also associate the specific sounds that are being made with specific movements, if the whales are also carrying biologers and those devices, their little suction cup devices, you can put on the whale, they're remarkable. It can detect the whale's depth, pitch, roll. It's so sensitive it can sense whether the whale is moving its flukes. So you have this amazing picture that wouldn't be accessible to human of exactly how the whale is moving and where it is when it's making different sounds. And then you can start associating these patterns. So what exactly is the whale doing when it makes this set of sounds? Oh, those are greetings. And what is exactly is the dolphin doing when it's making those sounds? Oh, those are echolocation clicks used for hunting. We can also decode individual vocal signatures that function much like names and then scale out and identified dialects because each family of cetaceans, let's say in the case of orcas or humpback whales, will have a specific dialect, a set of patterns that is unique to its family group. And that are passed down from one generation to the next and that do evolve over time, but that are a marker of culture and the algorithms can also detect those patterns. So because the algorithms are very powerful pattern recognition machines, they can then track those patterns in real time and give you a sense of what the whales are doing and where. All of this would be something that would take humans hundreds of years, but if the humans can label and interpret the data, which the AI, of course, cannot do, but once they sort of inject meaning into the data through labeling it, the AI can then automate pattern recognition. And what we get is this beautiful sense of the movescape, the sonic movescape of these creatures that helps us understand what they're saying to whom in what circumstance and when. And that's a building block for something scientists hope to achieve with AI, which is decoding animal communication. And if there is complex communication and something akin to human language, the breakthrough will likely come using AI studying species that are large brain highly social and long lived like whales or elephants. And indeed, there are a number of teams, you know, these really interdisciplinary teams bringing together computer scientists, machine learning experts, linguists, biologists to try to decode other species communication patterns. And I think we're missing some amazing breakthroughs in the next five to 10 years on this front. You're talking about AI as being kind of pattern recognition machines, but as you are saying there, they can also be pattern replication machines. So that gives us the ability to communicate, we think, so how close do you think we are to communicating with sperm whales using the dictionary that we've been able to build. So for context for your listeners, because this is a little abstract, imagine a zoological version of Google translate, where instead of options for, you know, Spanish or Cree or noctitude, you have an option for sperm, that is, you could type something in English and then sounds would be produced in the other language. Now, we have not yet invented that, but scientists hope that one day soon, we will. And that would allow automated playback experiments. So right now the methods we use to determine whether we understand the different vocalizations of other species are these manual playback experiments. So a good example would be we think we figured out what sounds female elephants make when they're in estrus and then some species they're only in estrus once every four years. So the males which don't live with the females are very attentive to these sounds and will come very quickly along distances. If they hear these particular sounds in the way you test that is you play that particular sound from a speaker in the savanna and see if the elephant males show up. You can do similar things for let's say the honey be alarm call for elephants the exhibit of very specific behavior, bunching all together, showering each other with dust. But those playback experiments are very difficult to conduct very very sort of slow and painstaking. Imagine if you have an automated device that could not only respond to human commands in real time to make sounds, but automatically pick up the sounds being made by these creatures and then play the appropriate sound back to them. And we are on the brink of being able to do that with bats and elephants and some species of whales. The problem is that we run the risk of exactly the trap the generative AI is posing for human societies essentially deep fakes of animal sound or sounds that sound plausible but actually don't carry the right meaning in the actual situation or context. So generative AI is anyone who's used chat TPT knows is prone to hallucination it will confabulate without appropriate guardrails it will simply mislead the reader because it cannot distinguish fact from fiction. So we would need all of those guardrails in place for these non human communication systems and those guardrails are going to be much much harder to figure out. So it may be that we have these automated playback systems and then we may do harm to these creatures we may mislead them it may just sound like gibberish so hard to know actually so I don't think we should overestimate our capacity to have complex conversations in real time with other species mediated by digital tech. We may be able to do simple things like issue better alarm calls or better interpret the sounds of other species so that we can ensure less interference at certain critical moments like nesting or mating but I don't think we're going to have a zoological version of Google translate available you know in the next decade I think that's much further off. As you're saying there are a lot of considerations to be born in mind on this what do you think are the most important guardrails to make sure that we don't mess this up. Thank you for asking that so the ability to record animal sounds and use this it to try to communicate with them seems very tantalizing you know a lot of people have a deep interest in nature and a sort of deep yearning to maybe. Understand whether there's species might be saying but there are great risks that I think we should be wary of including the potential for deep fakes precision hunting or poaching which would accelerate biodiversity loss particularly of endangered species so one of the guardrails I would like to suggest is that this is a technology that should simply not be used except by for the moment academic researchers who were subject to ethics review. There's too much potential for this to be used to lure elephants or rhinos for example by poachers there's too much potential to lead animals astray particularly we've got this huge problem of climate refugees at the moment as climate change is causing a lot of species to change their habitat and their range distributions. So my personal belief is that the first ethical guardrail applies to who should use it no one but academic scientists subject to ethics review not hunters and then the second thing I think is really really important is that we refrain from doing widespread generative AI playback experiments until we have a much better system in place for ensuring essentially an independent validation of the underlying dictionaries a lot of these tech you know innovations are kept private a lot of the harvesting of the data is being done by big tech and private companies it would be I think preferable to treat this data as a sort of a commons subject to scrutiny and I don't want to open access I mean a closed commons in which researchers can vet one another's data to ensure the animals are not being harmed the third thing that we need to take into account is indigenous data sovereignty much of this data is being harvested from lands which are the traditional territories of indigenous communities which have not given their consent for the harvesting of acoustic data and which those communities already have strong ethical guardrails in place deeply associated with their indigenous laws and relationship to territories and so their oversight of any experiments that might be undertaken is a third and very important ethical guardrail that I think we need to implement universally I think it reminds me a bit of a parallel debate that's going on at the moment amongst astrophysicists you have the search for extra terrestrial intelligence setty but you also have messaging extra terrestrial intelligence metty and some of the people in the metty world are debating who has the right to communicate on behalf of humanity if we're going to send messages out to the rest of the universe who gets to send those messages what should they be saying is that a kind of similar consideration you think into species communication on earth. That's a great analogy and it's a much harder problem on earth because the advent of this technology means many more people could be attempting these types of communication then could be attempting metty. So what I think will occur is a much needed overhaul of environmental regulations for example hunters are forbidden already from using some kind of bird calling devices during hunting season will simply get a refreshing update of those regulations on the one hand and then I think more broadly we need to take a step back and ask a general question about the implications of digital tech for environmental conservation because the issue you've pointed out with respect to the data actually applies across many fields not just acoustics it applies across satellite data wider data that we we now have a situation where the two big constraints on environmental conservation of the 20th century or being reversed by digital tech. So we now have an abundance of data rather than a scarcity of data and we have many people having access to that data rather than a few experts. So our environmental regulations basically date from the 1970s and we need to rethink our approach to environmental data in a digital world. There's some really good initiatives underway the United Nations Environment Program is putting forward an agenda for digital and environmental data that I think will create a broader context in which we can sort of systematically answer these questions for acoustic data but for also lots of other environmental data very intrigued by the commercial applications of this and in your book and whenever do you speak as well. You've talked about people using some of the information that we can gain from bio acoustics to apply that in the commercial world whether it's kind of honey bee algorithms that computer coders have used or whether to use sonic bionics sounds for cryptography and so on. Could you just tell our listeners a bit about that. What do you see people doing and in what ways are people imaginatively using the data that you're creating for commercial purposes. So there are many, many ways right now where low cost automated monitoring of non human sound could rapidly enhance biodiversity conservation really important because of the this massive unprecedented sort of mass extinction were experiencing on the planet. One of the examples I love is currently taking place off the east coast of North America in the Gulf of St. Lawrence where highly endangered right whales have been suffering through ship strikes traffic accidents and using bio acoustics to triangulate their location simply on the basis of listen to the whales we can figure out where they are at any moment in time and transmitting that information to ships captains in real time. And the ship captains then have to slow down stop move out of the way that bio acoustics protection system has led to a remarkable result not a single North Atlantic right will has died of a ship strike in that zone since this program was launched and it may be the thing that actually saves this species. So that if we generalize from that we can use bio acoustics to create these mobile protected areas that sense where an endangered species is on the basis of the sounds it's making and then constrain human action to make sure that we're not disturbing harming or entering or killing it. And so this new commitment to preserving biodiversity across the oceans means that we're probably going to see lots of these mobile protected areas in the oceans they sort of follow the fish as it were these protective areas just hovering over the species and diverting ships as they do so so imagine a future. It's not difficult to find out who it's easy to know what and where but why is harder. Why takes patience intelligence determination why takes over 700 of the best journalists in the world united by a fearless drive to unearth the truth. Why is difficult and more important than ever before for the why financial times.