The Truth About Autonomy

[Music] Welcome to the exponential view podcast. I'm Azim Azal, creator of this show and we are part of the HBR Presents Network. In today's conversation I'm going to be talking to Missy Cummings. She's an engineering professor and former fighter pilot and her expertise lies in human machine interaction and autonomous systems. These are both important issues that we're going to have to contend with as artificial intelligence in vehicles that's way into our homes, cars, planes and our lives. Before we get there let me just say how happy I am and I really am happy that you're listening to my podcast. You've all done an awesome job at giving it five star ratings. We've got many hundreds of them but I've also learnt that there's something else that really helps other people find the podcast and it's really simple. It'll take you 10 seconds. That is just hit the subscribe button on your podcast app whether that's Apple podcasts or Google podcasts or overcast or Stitcher. The best bit about hitting that subscribe button is that it'll help you too. Your podcast app will automatically find the next episode of the Experential View podcast when we release it. So please take a second and hit that subscribe button. Now if you want to stay in touch with me just sign up to my newsletter at www.exponentialview.co. That is www.exponentialview.co or find me on Twitter @azemazem. Well I'm delighted to be chatting with Professor Missy Cummings today. I first met Missy in 2018 and was hugely impressed not only by her work but also by her story. She was one of the first female fighter pilots in the US Navy. She flew the FA-18 Hornet at a supersonic combat jet and yes she's hurled herself off a 100,000 ton aircraft carriers and safely landed back on them many many times. From the military she moved into academia and she's currently a professor at Duke University and Director of Duke's Humans and Autonomy Lab. She has deployed her hands-on experience controlling $70 million of flying weaponry into her research on human technology interaction and automation. And this is a focus of our conversation today. Missy welcome to the Exponential View Podcast. Thank you for having me. So we have so much to go through but I'd like to start by laying down the premise of your current work which is that technology can and will automate many tasks but we still need to keep humans in the loop. Can you expand on that a little bit? That's correct. So the name of my laboratory at Duke University is the Humans and Autonomy Laboratory and there is an acronym there how and it is a takeoff on space Odyssey, the computer in 2001 that goes rogue and tries to take over from humans and I like having that play on words because you know that's really what we're here to prevent. The mission statement if you will in my laboratory is humans and automation or autonomy need to work together to achieve a greater outcome than either could achieve individually. It's quite a visionary description but I think if you talk to the person in the street about autonomy and autonomous systems today we'd probably be thinking about autonomous vehicles or you know self-flying drones. Where are we today with the reality of autonomous systems? So autonomous systems show up in many many areas of life not just in transportation systems. I think for your average person they hear about the most through this idea of driverless cars or self-driving cars and those are still a future possibility yet not a reality. More distant future are the flying air taxis, flying cars that people like to say but there's also significant work today to look at autonomous medical systems for example and smart systems in your home that potentially can change your thermostat or maybe do security for you. These would also count as autonomous systems. So what are the commonalities that we find between things as different as a thermostat and a flying air taxi? Are there commonalities? You have to ask yourself what part of the system is being controlled by a computer and what part of the system is being supervised by human and if you have human supervision of a computer and then that computer actually executes control somewhere in the world that is an autonomous system and so if I pick up my phone and tell my phone that I want it to heat up my house before I come home that is an autonomous system because you're remotely commanding it to do something for you. That is actually exactly the abstract process that happens in an aircraft. In today's aircraft commercial aircraft that you fly in the pilot tells the aircraft I want you to go to 30,000 feet and maintain 300 knots and then the aircraft itself figures out how to do that. We have increasingly got these systems that figure out what they think we want from our environment. So I think my thermostat doesn't really ask me many questions and I don't give it many commands. It sort of figures out through the weather and our behavior what we might want and therefore how to run the boiler and the radiators. This is a huge topic of research going on right now. This idea of intent recognition and this is where you see artificial intelligence being infused in systems. Today's smart home management systems really are not that smart. I mean they may be able to pick up a pattern of what time you come home and change the settings based on a typical pattern but that first day that you come home early or you go on vacation and then you realize you actually have to re-expressure intent to the system and in fact we find that frustrating especially when talking with maybe our smart phone assistants like Siri or Alexa we're constantly having to correct these systems because they actually do not get our intent correct. So our experience with these systems is perhaps slightly less than what the manufacturers have presented and it seems to me that right now we might feel a little bit gloomy about automation. We've had a couple of crashes of state-of-the-art aircraft that Boeing 737 MAX and we've had a number of Tesla fatalities. How should we interpret what's what's going on there? Well I think your average person needs to realize that this problem is hard and we're still in the early days of figuring out how to design truly effective and safe autonomous systems. I think the interesting thing that most people do not realize is that the autonomy, the underlying computer code for example that's in a Boeing 737 MAX is basically elementary school level automation compared to the collegiate level automation that's happening inside of driverless cars. And so the code is much simpler in aircraft and in fact we in the 737 MAX case we should have been able to catch that problem because it's a known problem and we know how to test systems that have what we call deterministic code. Code that's not based on probabilistic reasoning. But in driverless cars there's a substantial amount of probabilistic code that's making guesses and no one knows how to certify these systems. And so it's substantially harder to test a Tesla or any self-driving car to make sure that you know what it's going to do. Right so there are two different types of systems. You've got these deterministic system which effectively follows a set of quite complex rules then you've got probabilistic systems where outcomes are much more uncertain and the information that's coming in is much more uncertain. So perhaps let's turn to driverless cars first and maybe if we have time we'll talk about the aircraft as well. There are a lot of claims made by many of these driverless car companies that they'll have fully autonomous vehicles that can drive themselves under any conditions on the roads within a couple of years. As you look at the field do you think that's likely? No. I think that it is a lot of interest in driving market share prices up and generating a lot of hype. But we are simply we being the driverless car community autonomous systems engineers. We're not even close to developing what we call level five cars which are cars that can go anywhere anytime under any conditions by themselves and you being a passenger in the back seat. So no that claim is completely false. The reality is that we will be able to achieve and in some cases have already achieved limited forms of self-driving. So there's a slow speed shuttle for example in Las Vegas that you can take and and that's been pretty successful in a very limited domain. And so I actually think it's very sweet when we look at the human desire to move into the jets and jets and age. You know we want driverless cars and we want self-lying cars because it enables us to have a new level of autonomy in our lives and freedom and control and just share coolness that we have this kind of technology. And I think that that dream is critical for moving the football down the field. But then I do step in when I start to see people make claims that I think really impenetrable. and John public safety. And that is actually where we are now. - On the journey to this Jetsons future, there is a very widely accepted hierarchy, sort of ladder of autonomy. Could you just refresh us on what the different steps are on the way from my having to drive my car or fly my plane if I could fly entirely myself through to the thing doing it for me? - There are across a number of domains, things that we call levels of automation or levels of autonomy. It's this idea that we move from level one, which is humans doing everything to some higher level, some scales are five, some scales are 10, but in the highest level, computers are doing everything and humans are just sitting along for the ride. It puts everybody on theoretically a same sheet of music. So I think that that's why these levels and discussions about these levels have become so important. So in driving, there are five levels, five being complete driverless car, you just can fall asleep in the back of the car and it'll take you anywhere. Level two is driver assist. The move from level one to level two, we've obviously done it and it costs a lot of money. And let's just roughly say that's taken 10 years and a billion dollars. The jump from level two to level three, which is partially automated under some conditions. Okay, maybe another 10 years and a similar amount of money. The jump from level three to partial automation to full automation of level five is exponentially higher. It's not a linear relationship, not in time and not in money. At this point in time, it is unknown how much money is that it's going to take to get to the level five. We have no good estimate. It's billions with a big question mark at the end of it. Yes, I've spoken to a couple of people on that subject and they come out with numbers at range between 20 and 40 billion dollars per vehicle platform to get to level five. The thing that's always struck me about those levels and you made the point that it's the non-linear moves from one to the next. Level three always struck me as quite a curious level. It's the level where you can let the vehicle do its thing but you need to be ready to take over sort of at a moment's notice. And I always wondered about that because I was thinking, if I'm letting the vehicle do its thing, how do I keep myself in a state of preparedness to jump in if it goes off the rails? Level three is a very difficult. I think of all the levels it represents the most dangerous level that you could achieve because of this idea of the hand over back to the human. I actually am against level three as a concept with a couple of exceptions because humans generally are terrible at monitoring a situation and then being expected to take over control at a moment's notice. Inside every human is something called the neuromuscular lag. So you have about a half-second delay programmed into you by nature. You cannot get around it. And so if something happens and needs your response within that half-second neuromuscular lag that you cannot get around, then it's simply impossible for you to react in that time. And there are a lot of situations in driving that fall within that category under very limited domains where if you do not take over the car can either bring itself to a stop or pull over to the side of the road that gives you some kind of safety guarantee. You know, so I think those are permissible quasi-level three areas. But in no way, shape or form should humans be moving at highway speeds and then be expected to be given control. And we've already seen multiple times that that can end in death. The idea that a car should be able to deal with the driving conditions in Phoenix, Arizona, Jakarta, Indonesia, Rome, Italy seems pretty extreme. I sometimes wonder whether in order to really support autonomous vehicles we need to build smart infrastructure in our roads, whether it's sensors or signaling systems that help the vehicles figure out where they are and what to do next. So there's a lot packed inside what you just asked. Let's take the teaching concept and then we'll take the infrastructure concept. Sure. So I think that it is such an egregious use of the word teach and learn when we start talking about autonomous systems because what we're really talking about is artificial intelligence and even inside academia there are very heated debates over what it means to teach and what it means to learn. In our world where we're teaching and learning, teaching humans and expecting them to learn, ideally we teach them an abstract concept and then people can take that abstract concept and apply it to multiple different situations. And so when I explain to someone that you need to look for if there's a ball that rolls out in front of someone in a car that you need to look around because there could be a child attached to that ball. That's a classic driving example that people are taught in driving school. The reality is we're not teaching okay every time you see a ball you must associate that with a child. We are actually giving people an example of there could be something that happens in front of you that has other events associated with it that you can't see but that you just must imagine. And so the case where you give Jakarta or my favorite driving test intersection is in India where there are trikes and motorcycles and people and cows all moving through an intersection at one time. So driverless cars using the machine learning techniques that we have today must see countless examples, millions of examples of every single scenario in almost the exact way to quote unquote learn from it. One thing we know for sure is a half inch of snow on a stop sign will cause an artificial intelligence algorithm to not see a stop sign anymore just because there's a half an inch of snow on it and it can't abstract away the fact that there's a slight environmental change. And so in this way this is why the systems are so brittle and why I've been calling for a vision test for these systems because we know we being researchers inside the system know that the cars really cannot adapt and cannot derive the abstract principles like humans can. I'm not saying they never will but they certainly don't right now. The approach that we take today with teaching these cars seems to be that it is purely empirically driven. So there's not much in the models that is a sort of a prior I set of abstractions rules entities relationships between them it we're building off the breakthroughs and machine vision of the last decade. And I just want to slam all my colleagues in artificial general intelligence which is a big chunk of academia. If I hear one more time that we can derive all the principles of the universe if we could just have a big enough neural net to run everything that simply ludicrous and anyone who tells you that we just need more data to solve all these problems is wrong because we don't even know what those sets of data would actually need to be to cover all the same bases that humans cover through abstract learning. So I wanted to address the issue separately of infrastructure so if we agree that the artificial intelligence machine learning algorithms are brittle and necessarily limited and can only go so far I do think it's an a reasonable approach then to say well maybe we should augment the infrastructure in our systems to help those cars fill in the gaps where they can't reliably reason well enough. So I do think that's a good potential outcome the problem with that is the expense of any system goes way up. Right so you could imagine high traffic areas having these shims in place that make it easier for autonomous vehicles and the low traffic areas wouldn't and you would need to rely on you know the black box inside my brain to control the vehicle. One thing I'm curious about is that the relationship between the human and the autonomous vehicle is really one of substitution right the autonomous vehicles brain substitutes for me and that's that's the vision when I look in some other domains you know chess or go or even some medical diagnosed diagnostic areas there is much more of an idea of human plus machine that the machine is a little bit like a superpower to a human and getting them to achieve more and achieve better outcomes and when we think about the transportation domain is there is there any analogy that helps us think through man plus machine being more than just man or machine. And driving since really bad things can happen in less than half a second, there really can't be effective man machine in interactions that require joint control of the car. It's either the human or the automation because if the human is not perfectly paying attention, there's no chance that the human can intervene. And even in many cases, when we're in an accident, when you start hydroplaning, for example, that's out of your control, that accident is happening regardless of your ability to have some kind of input. And so I think where we see more effective man machine interactions are places where you have a little bit longer time scales, which includes aviation. So it seems counterintuitive, but aviation is actually an easier domain to automate than driving because generally you don't have 50 other idiots within inches of you also flying, right? I don't be right. - Well, I'd love to turn our attention to flying now. About 10 years ago, I remember watching on TV just something so remarkable, which was Captain Salenberg, a Sully Salenberg, a landing, that plane on the Hudson River. What do you think we would need to do to have an autonomous system that could walk through his rapid series of judgments when he started to realize that he had some kind of power problem on his aircraft? - Well, I'd like to take one little step back and say, before we delve into can we automate that today, the reality is that the miracle on the Hudson with Sully Salenberger is actually a great example of human machine collaboration primarily from the engineers of the Airbus aircraft. There is actually a button on the aircraft that says, "Ditch, and when you hit this button," which he did, the aircraft was pretty smart, i.e., the engineers that designed it were pretty smart, to seal up all the external ports and actually change the flight control system slightly so that it knew when it would be coming in for a landing, that it would have smoother landing characteristics across a body of water. And so humans, in that case, thought very far ahead and then the aircraft helped him make a much better water landing than probably would have happened if he'd been in any other aircraft. So in that way, it was a collaborative maneuver to help him be, and he's an excellent pilot. And I don't want to take that away from him, but the plane itself was designed for just that environment. Now, if we look forward and say, could that be automated? And the answer is yes. And people have actually been working on this problem for at least 15 years about how to engage the automation to do emergency landings. The US military has head drones do emergency landings for years without your knowledge. And so it is very possible for aircraft to land themselves in extremists without a human in the loop. Now, we haven't done that with passengers on the plane. That's an entirely different bar. And the FAA is in no way going to certify that anytime soon. But the answer is it is possible. It's just going to take a lot more money and a lot more time. And the airlines themselves have decided, you know, is this really something that we want to put in a plane? Because in the end, people have something that we call shared fate that really drives them to want a pilot on the plane. And shared fate is, you know, you're OK knowing that the automation is doing most of the flying because you know there's a human up there that shares your fate and is going to do everything they can to save his or her life along with your life. So I do think that there's psychological component of wanting a human on the plane really more as a security blanket. Even though planes, for the most part, do fly themselves today, I don't think we're going to get rid of the pilot in command, probably ever on passenger flights. Interesting. The idea of the pilot is having skin in the game or perhaps in the early days of elevators and buildings. There was somebody who would sit in the elevator as it rode up just his job was simply to look calm. So to prevent panic. We talked about-- we're talking about aviation. And as we record this, this conversation, we've-- we're on the back of a couple of pretty horrible fatal accidents involving state-of-the-art Boeing 737 MAX. And I think a lot has been written about, you know, what were the underlying drivers of those accidents? And I know we don't have official reports yet coming back from the safety authorities. What do you think the issue within the 737 MAX could be? I would say number one, there were clearly flaws in the software code. And those were not picked up in traditional testing. Boeing engineers are good people. They did not want this to happen. They were not intentionally missing work and trying to do a bad job. They just simply missed a set of conditions that, in theory, we should have tested because we know about the air France crash off of Brazil when something very similar happened. But I would actually say, as an engineer, the bigger problem was that the testing protocol did not get that. The reality is software code errors happen every day all the time. And you need to develop systems that are robust to errors in the code because we know it's very, very difficult to go find every single error in every single line of code. But then even more important than that was the fact that the pilots did not have the right information that they were presented with. And this is this idea that safety is for sale. The reality is airlines all over the world can pay for different training packages. Forget the hardware. Right now, certain airlines get more training than other airlines because they decide to pay for more training. And so the reality is that we know the more training makes a airline safer. So should they be telling you that as a passenger? I'm sorry, we only got package B. We didn't get package A. We got the least amount of training that Boeing deems is safe. And what is safe? We don't really know because the regulatory agencies around the world, I think they model maybe what the FAA does, but they don't have to act like the FAA. Beyond even that issue, what most people don't really realize is the 737 max is not a 737. It is actually an entirely different aircraft. And Boeing called it the 737 max. It shares some similarities, but it is not the same aircraft. But by calling it the 737 max, it actually was able to piggyback on the certification of the 737. And so the reality is the messy truth is, it should have gone through a different certification process than it did. And so this is a bigger problem, which of course implicates the Federal Aviation Administration in the United States and also regulatory agencies in other countries. That is a fantastic and densely packed answer, which I'm going to encourage listeners to listen to again. What you've actually identified is is several different problems operating at different scales in an overarching system, which then I think demands much more vigilance and engagement by the public in different guises, whether it's through our regulators or through other entities, to ask questions about the systems that are actually being built. How do we need to think about changing those certification systems as we move into a world of more autonomous systems? And where are we going to find the talents to manage them? So I think this is a great question. It's probably one of the most important questions that I think anyone can ask at this time in history. And if you go to my website, I've written a paper about this, about the problems of certification moving forward with autonomous systems. Because at least in the United States, this idea of equivalence is how systems are being certified today. And it's not just aviation systems. I actually think-- and I'm being very controversial here-- and this is just my opinion-- I do not think that robotic surgery in America is safe because it was certified through an equivalence procedure by the Federal Drug Administration to say that it was equivalent to laparoscopic surgery. And so because robotic surgery is equivalent to laparoscopic surgery, it was basically quickly shepherded through the certification process in America. This idea of equivalence has shown up in the 737 issue. The idea of equivalence is now showing up in driverless car systems. Well, I've got this driverless system. It's been performing pretty well. And since the National Highway Traffic Safety Administration in the US doesn't actually regulate until a certain number of people die, and they won't actually tell you what that number is, they're basically saying, well, it's safe enough until you get a certain death count. And then maybe we'll go back and review. If it was safe enough to be on the road for some period of time, then that's good enough for us. So I do think that regardless of where the autonomous system sits, if people think that if they do an as-if comparison, it's wrong. They are new systems. They are incredibly different than anything we have ever seen. before. And especially given the fact that no one in the military, no one in the commercial sphere of any domain understands how to certify a probabilistic system, then the public really is at risk in being exposed to any kind of system that includes any kind of probabilistic reasoning because we do not know how it's going to perform. You may be thinking about something that happened around a decade ago, which was in the financial industry, the banks came up with all sorts of very exotic synthetic financial products, credit CDOs and CDOs squared they were called. And at the time regulators didn't have the ability to stress test what happened to those in extreme circumstances and they essentially relied on the models the banks themselves had created to sign off their own products. And then of course what we discovered and we've all lived through since 2008 through for that two or three year period was that those products then rapidly blew up and exploded metaphorically across the global financial system. So it seems to be that we have at least some recent history perhaps in different domains of what happens if you allow the research and the the the march of technology run very far ahead of the regulatory systems. I think probably the biggest problem faced worldwide but is particularly acute in the United States is our regulators are not the top performers in artificial intelligence or any kind of probabilistic reasoning system. And I don't mean to come down on them but at least United States our regulatory agencies typically are old older in demographics. They have no experience, zero experience in coding systems for example or managing systems that have a lot of code in them. And I would even question any people who could actually look at a probabilistic system any kind of autonomous system and have any clue whether or not it's performing to any satisfactory levels. And so if the C team is going into regulatory environments to be regulators but they have no experience at all in these systems then what I see is the perfect storm coming which is a lot of dangerous systems being put on the market without any real way to stop them. And I think the failure of IBM's Watson is a great example of this. So IBM Watson built on the same algorithms that do jeopardy and everybody think wow that's amazing look jeopardy the Watson and jeopardy does great why was there such a catastrophic failure for Watson when it came to medical diagnoses because it's brittle because it can't do abstract reasoning. There were and maybe still are many lawsuits pending about this because of the true lack of performance in these systems. It seems that we need we need two things don't we we need some fundamental breakthroughs in AI development that allow these systems to deal with with Brickleness and we're not there yet and it sounds like we also need to find regulators who love Bayesian reasoning and stochastic processes and entice them into those roles. I agree and I have put several proposals forth in the United States of the National Science Foundation to actually start training regulators and policy makers in tandem with engineers. Sadly my ideas have not yet taken hold in the funding agencies and so you know I I beg the community listening help me on this one because maybe the government agencies will understand the importance of this if they get enough pushback from communities that indeed they do want regulators who know what they're talking about. I think one of the challenges here is the cultural issues and particularly the cultural issues within the technology industry and the technology industry that comes out of Silicon Valley where there is you know there is a view that in a sense all regulation is is unhelpful and unhelpful for progress progresses this thing in of itself and and you've written a little bit about the the culture of Silicon Valley and how it compared to some of the experiences that you had within the military. Why did two different cultures like that have exhibit similar behaviors and what are those behaviors? So there's been a lot of debate lately about why we can't get enough women in Silicon Valley. These environments are all male almost all male definitely predominantly male and they the managers inside these environments encourage or at least do not stop what I would consider fraternity like behavior and the development of anti social behaviors that are very off-putting to women and so I've written an op ed for CNN about how the work culture in Silicon Valley is still very sexist, very misogynist in in California that in the year of 2019 looks a lot like the 1990s when I was a female fighter pilot to the extent that you know when I was a female fighter pilot it was not unusual for everybody to go to the strip club to debrief their flights. I don't think they're doing that anymore so in that case the military has actually had an improvement but it's still happening in Silicon Valley where lunches are happening in strip clubs in Silicon Valley. If we're going to continue to have these anti social climates that continue to cause women to leave then this is just going to be a continually self-circling problem that's not going to get any better and so I think management needs to step up to the plate and start taking very specific actions to stop these kinds of behaviors. As an engineer who builds and analyzes this technology, when you look out over the next few years do you find yourself being predominantly optimistic about what the near future holds for us or are you pessimistic about it? I'm a techno realist so I try not to to be too pessimistic. I mean I'm an engineer. The reality is we fail all the time. If you walked into my lab right now you it would not look like a drone laboratory it would look like a drone graveyard because they're crashed everywhere and they're in pieces everywhere that's just what engineering is especially in research and development and so I am very optimistic driverless cars. We will get there. We are just not going to get there in a year or two years or really even 10 years but we will get there. Will you get a flying car? Yeah we'll get there. I'm not going to be alive to see it sadly but we'll get there and so my advice to everybody is take a deep breath go for a walk you know enjoy your life and it'll get here when it gets here but you know you need to set your expectations in reality and also hope that the you know push for government change in concert with technology change because in this case if we're not on top of these technologies and we've certainly seen this with Facebook and other social media technologies then what we find is that we're trying to get that horse back in the barn after it's gone a couple hundred miles away from the barn. Missy thank you so much for sharing a healthy dose of technorealism with me on the exponential view podcast today. You're welcome. Well thanks for listening to my conversation with Missy Cummings. To stay in touch follow me on Twitter I'm @azeme that's AZ EEM and subscribe to my weekly newsletter exponential view at www.experentialview.co. I'm your host Azim Azar this podcast was produced by Maria Gavrilov and Fred Casilla boy in Sabia cellar is a sound editor exponential view is a production of e to the pi i plus one limited and it is part of the hbr presents network.