Waymo's Dmitri Dolgov: 20 Million Rides and the Road to Full Autonomy

We have an unbelievable treat next, a founder who's touching a ton of lives and who's been at it for a very long time. How many people here have been in a waymo? Okay, well it's a relief, this chef eats his own food. I do, I do a lot of it. All right, and how many people love the waymo experience? Rock out, I'm a daily active now, it's incredible. Excellent, thank you. We have here, the creator, a man who has been at this mission for Get This Founders who have been in AI since 2022, almost 20 years, building in the autonomous vehicle challenge. And he has not only been at this, he's been at it in the great times, at the tough times, he's been persistent, and he has created something that is unlike anything else on Earth. Thanks for the exceptional, please join me in welcoming Demetri Dolgoff. Thanks. Ready to be here. All right, Demetri, so we've got about 25 minutes together. The goal is to understand a little bit about you, what makes you tick, what has made you persist since the early days of the DARPA challenge 21 years ago, all the way through waymo early days to today and the future. Let's start on you and then we'll get into technology very quickly. Sound good? Sounds good. So Demetri, you are known by your team as technically brilliant, incredibly intense, but also very kind and humble. You were born in the Soviet Union, raised in the states, and then chose to go back to one of the most prestigious and tense physics programs on the planet in Moscow. How did those first few years of your life shape you and how did it shape your character? So my parents went to the same school so that to a large degree drove my decision, to go back. I went to high school, actually traveled around quite a bit. I spent a year in Japan, then went to high school in the states and came back to college to do math and physics in Russia. So that was the same school that my parents went to and I kind of really grew up hearing stories about what it is like to be at that place. I really wanted to go back. And I think in terms of how it shaped me, I think it really set the foundation, you know, technical foundation and the, you know, in those early days in college, one of the most important thing is, you know, acquiring the ability to learn and independently explore. So I think that really helped me in my future career. Now you did this very intense program at Moscow Institute of Physics and Technology and then you decided to keep going on the AI path, you earned your PhD also in AI. And then you pretty quickly were attracted to autonomous vehicles. In '05, you were a part of the DARPA challenge. Can you tell us about those early days, what drew you to autonomy? That was kind of a light switch moment for me. I, you know, in the early days when I went to college and I went to grad school, it was more about just the learning of the fundamentals and I didn't have at all a clear picture and idea of what I wanted to do after, all right. And then I think the timing just was just incredibly lucky that this was when I was finishing of grad school, the urban challenges, the ground challenge and then the urban challenge the one that I took place in were happening and it clicked. It was just the technology is incredibly interesting. The mission is so powerful that nothing else comes close and it's a real product there. You can be hands-on, experience it yourself. So it really checked all the boxes for me. And as you said, there's been 20 some years ago, you know, who's counting and then I've never looked back and that's what I've been doing since. Amazing. So Waymo started out of a project at Stanford Automotive Lab. There were two sides of this building. There was the autonomy side and then there was a solar car side. Fun fact, I was an idealist, I worked on the solar car. I got that bet very wrong. You bet on autonomy, tell us about the first few years of Waymo from '09 to the formative years. So we started in 2009, that was at the time the Google self-driving car project. And the first couple of years, it was all about learning the problem space, understanding what it means to try to put a autonomous vehicle on public roads. So when we started in service of those goals of learning, understanding the problem space, we created a couple of goals for ourselves. One was to drive 100,000 miles total in full autonomy, which at the time was not heard of. And the second one was to drive 10 routes. Each one was 100 miles long, they were all over the Bay Area, chosen to be very difficult. And we had to do each one from beginning to end in full autonomy, still with a person behind the wheel that can take control. But the challenge was to complete each one without an intervention. So it was a small team of us, it was about a dozen people. It was the early crazy start-up days, everybody working 24/7, riding code and building hardware during the day, then doing some testing at night. And it took us about 18 months to complete both of those challenges. Incredible. It seemed impossible at the time. How you guys are on hundreds of millions of miles? Absolutely. Okay, so early Waymo days, extreme challenge, starting to achieve their next few years. You have a reputation in your team for grinding really hard, you're sleeping at the office. Tell us about Dimitri in the first few years of Waymo and how you formed your leadership style. I gotta say, those early days are probably the most fun I've ever had in my professional life. And it is that momentum and that pace of an early start-up days when you are making so much progress every hour of every day. And you're doing everything. You are working on setting up the hardware in the cars and then configuring and calibrating the sensors and your post-estimation system. And then you are riding software during the day and it's everything. It's the core of the software, the algorithms that drive the car. It is all of the tools and UI and the user experience in the car. So you're doing everything. You're learning at an insane rate and you're making progress at an insane rate. That was those were the early days of Project Shaffer. And then in those couple of years we've convinced ourselves that, yes, this is worth pursuing. So we doubled down and started actually building towards the future of a fully autonomous product. Okay. So exciting first few years, intense, fast-paced, technically really difficult. Now take us to 2016-17 for a moment. This was a period where we actually had a hype cycle in AI. Turns out there's been a few of them. But AV, autonomous vehicle, was at the center of that hype cycle. I mean, I remember just so many companies going after this. And then there was a massive slump. And when most people gave up or failed or fell apart, you guys persisted and you were a leader in that persistence. For all the builders in this room, how did you navigate through the hard times? So first the comment on what these cycles kind of look like to me and how I've seen them and you said there's been many summit in AV, but more generally. And kind of what often leads to a cycle like this is some rapid, some breakthrough that leads to very rapid progress in the early parts of the problem. And the AVs, the problem has always had this property that is very easy to get started, but it's very difficult to take all the way to a real product full autonomy and superhuman performance. It's somewhat natural given those ingredients that whenever there's been big breakthrough in technology, whether it's convolutional nuts or transformers or large language models, it's led to this cycle. OK, now the problem is going to be, and it kind of restapes the early part of the curve, but it doesn't change the long tail of it. So for us, I think it was understanding that it's not going to be an easy problem, but it's a very important one and believing in the mission. Today, worldwide, somebody loses their life to crash on our roads every 26 seconds. So I guess it's the combination of knowing that the mission is really, really important and then understanding what you're up against and not looking for easy wins or quick solutions or silver bullets that help the team have the right stamina to go the distance. Brilliant. So you guys were in this moment where it was really easy to get started. A lot of people got there, but you guys actually persisted and got through to the other side with a truly magical experience. Pretty much every hand in this room went up, a truly magical experience because of that persistence. Let's talk about technology today. A lot of people are talking about world models. You have had all the components of world models for many years. How do you think about a world model and what is Waymo's version of of a world model. Yeah, there are a few things that a few terms that people use nowadays. People talk about world models, world action models, omnimodels, visual language action models. And at the core of each, there's an ingredient that is relevant and really important for Waymo and for what we've been building in our AI ecosystem. At the core of our AI ecosystem is what we call the Waymo Foundation Model. And it powers three main pillars of our AI and our tech, the driver, the simulator, and the critic. And those are very related but distinct tasks. So at the core of what our foundation model needs to be capable of are things like it needs to understand how the world works. The physics, the dynamics of the physical world. And it needs to understand what it is to be a good driver and how the effects of the actions of that driver or I agent on other agents in the world. And then we need to instantiate those in the physical agent that we're putting on the roads. So in a way that foundational model that we've been building over the years is a multimodal world action language model. It's this multimodal in that it needs to be able to reason about not just images or video but also other sensors like lighters and radars. It is a world action model in that really has to have a deep precise understanding of the 3D spatial properties of the world, the dynamics, the physics, the behavioral aspects of other agents like cars, pedestrian cyclists, and so forth. And we are not just passively modeling those worlds. We're an active participant in it. So we not only have to-- the world model has to be controllable, but also we need to have a deep understanding of what it means to be a good agent in that world. And finally, it's aligned with language. And that allows us to pull in the general world knowledge of a VLM into our model that is very, very useful in giving us a boost in understanding the semantics and the deep social aspects of driving. And we've been working on productionizing that model for years. And it really requires an extremely high degree of performance and accuracy and realism in every aspect of what we just talked about. Brilliant. So with this driver simulator critic architecture, there's also been a lot of conversation about end-to-end architectures. Is that the appropriate economy? How do we think about the approach to getting us to extreme performance efficiency, autonomous vehicles that are totally generalizable? That would be very clear. The world model that is described, the wave of foundation model, is an end-to-end model. So when we talk about an end-to-end model, typically mean that it's one model that goes from sensors to decisions or actions. And this is a very nice properties of such a model. One of the most important ones is that it learns the right reach representations between different components of the system, like the encoder in the decoder or the perception and the planning part of your system. As opposed to something where that interface is engineered, which is not sufficient for a task like driving. Now I do think there's a false dichotomy there. There's end-to-end or something else. Really, my mind has always been the question of it's end-to-end, and then what else? And what else do you need to build if you want to have a product that is fully autonomous, has superhuman level of safety. And you want to deploy at a scale and drive hundreds of millions of miles. And there turns out that the basic vanilla end-to-end system is insufficient. So there's a massive difference between using end-to-end versus purely relying on it. So at Waymo, we've really gone beyond that basic vanilla end-to-end approach. And we've augmented the learned representation with structured materialized intermediate representation. And what that allows us to do are a few very important things that you might not actually need if you are building a different product, if you are building a driver's system or a prototype of demo or a small scale deployment. But again, those things are absolutely critical if you want to go all the way to a fully autonomous safe system with superhuman performance. And those are things like having extra validation at runtime of the agent that's running on the car in the physical world. It's things like richer training and evaluation recipes that are very difficult or impractical to do in a pure kind of basic end-to-end system where this structured materialized representation gives you a boost in things like closed loop evaluation, closed loop training, richer reward functions for reinforcement learning, so that's been our approach. - And all the human feedback that you get from support and drivers dropping in and all of that, it's essential to have this type of architect. - Exactly, exactly. - Makes perfect sense. So not only have you innovated on the software stack, but also the hardware stack, there's a sixth generation now of Waymo Driver. And you guys have always focused on being the driver. Tell us about the new sixth generation and what was it like the first time you've interfaced with it? - Yeah, so the sixth generation is our most advanced hardware suite and says the suite yet. The focus there has been on performance but also on simplification, drastic cost reduction and high scale volume production. And this is the driver that's powering our latest vehicle platform, that's the all high. We earlier this year started fully autonomous operations and it's currently only open to employees but coming to all of the writers later this year. And I had a chance to take a ride in one, as soon as we started running fully autonomous operations. And I spent a lot of my life in various generations of our cars every once in a while. There is a new first moment and that was definitely it. It's just that the coal cars designed around the writer experience even though the extra footprint of the car is about the same as the eyepiece. But inside you get in it feels like it's the living room. It's so much space in the back, we have new screens, we have these doors that slide open and open automatically when you approach the car. So I had a blast and I can't wait to have this car in our fleet open to everyone. - So you guys are going through a period of incredible scaling. For many years you were in the lab R&D purely. It took 16 years-ish to get to 100 million miles, six months-ish to get to 200. Things continue to scale really rapidly. 11 cities now, many, many more on the horizon. Tell us what is it like to scale a new city? And then tell us about your daily life with AWAYMO. How do you use it as a creator? - Well, there's a lot, okay, it's the exponential scaling. First of all, absolutely, it's been a face transition for us and how we're scaling. So to give you a couple of additional data points, it took us eight years from the day when we started our fully autonomous operations to the day when we had our service, our driver, providing rise to the public in four cities. Earlier this year, just a few weeks ago, we launched four cities in one day. We've given over 20 million fully autonomous rights. 10 of those million happened in the last seven months. - Amazing. - So that's what exponential scaling looks. - Amazing. Launching new cities, there's operational components to show up, you have to collect the data, characterize the environment, validate the driver, a significant part of it is starting the conversation with the local communities because it's a new thing, it's a new product. So it's on us to earn the trust of the people there. And then more often than not today, we're seeing that the driver is generalizing incredibly well and is just a matter of high fidelity, rigorous evaluation and validation before we deploy the fully autonomous product. And then we go from there. And then the last part of the quick, oh, what is it? - You're doing a lot of data. - It was a multi-part question, yeah. Wait, I mean, Waymo is how I get around nowadays. That's how I got here today. I was like, you know, a great, right from Palo Alto up to San Francisco on Freeways. My family used it. I have three kids. They love Waymo. They, I think nowadays they get annoyed if on a rare occasion we have to be in a car that's driven by myself or my wife or another human being. They're like, okay, what's going on there? - I feel the same way, it's the same way. - They love it. It's been part of their lives for the entirety of their lives. There's when we are driving around, there's two things. There's only two things that get cold outs from my kids nowadays that's doggies. And it's Waymo's. - Nice. Okay, probably some more amounts of cognition between those two. Okay, so let's talk about safety. You know, one of the most meaningful, exciting, getting parts of partnering with Waymo has been the fact that there's 1.19 million people a year on Earth that die in road accidents. It is, this is life or death, not only does it touch everyone in this room, but everybody has some connection who's been impacted by this. You have been about safety from the very beginning, and it's actually pretty hard in a Silicon Valley where it's moved fast and break things and see what happens. You guys have been incredibly patient with safety. Can you tell us about a story that made it very real to you and how you keep that safety culture at Waymo? The numbers you've mentioned, that's what drives all of us at Waymo, and the status quo is not okay. We've kind of grown this over time, but challenging the status quo is really important to everyone at our company. You're absolutely right that how you go about building a system like this is different from you might do in other areas, in other fields, in other industries, where safety has to be the non-negotiable foundation, and you have to build that into your, everything that you do from day one, your model architecture, your training and evaluation recipes, the mindset of the team. You can be very tempting to focus on capability first and get to the 90 percent very quickly, but how you go about the first 90 percent is a totally different problem for how you go about getting to your next N9s. So keeping that in mind and focusing on safety as the non-negotiable fundamental layer from day one is super important. And then today we're driving more than 4 million miles in full autonomy per week, and you see a lot of events from the field. And today we have the data over 170 million miles, full autonomous miles, where we see that the Waymo driver is more than 13 times safer than a human driver when it comes to serious injury, causing collisions in cities where we operate. And you see that sort of super human safety behavior, manifesting itself on the roads daily. I see examples of recent, there was a little while ago, there's an example that I saw of a person, I think he was a young woman, an electric scooter on a road, and then she lost control and tripped and fell right in front of the Waymo. And the Waymo driver showed super human accuracy and reaction time. I was able to serve and break, and everybody walked away. So it's things like this that I myself personally and the team, the whole team find very rewarding in terms of actually having a real impact on the safety of our roads. And the scale that we're operating, that 13X reduction means that we are preventing a serious injury every eight days. And that impact will just grow as we scale up. >>Wow. >>We're going to open the room to audience questions, a couple in just a moment, before we jump in. I heard a story about the LiDAR detecting or the radar detecting the footsteps of somebody behind a bus. Did that happen? And how does that work? >>Yeah, this was one of those moments where I was positively surprised by the emerging capability of our system. And the situation was, this was I think in San Francisco, the Waymo driver was at a intersection, there was a bus that crossed, and we're sitting there waiting at a red light, so the bus crossed and stopped partially blocking the intersection. So our light intersection. So then our light turned green, the Waymo driver started to proceed. And as it's proceeding, it detects a pedestrian on the other side of the bus. And you can see through the bus, it's not through lighters, not radars, not cameras. The windows are reflecting that people inside the bus. And then it starts to react defensively, and sure enough, a pedestrian emerges from behind the bus, and then we're able to nudge around them, and everybody goes on their way. So when I saw that, I was out of it, blew my mind, I'm sure was going on, capable as superhuman as the Waymo driver is, it doesn't see through solid objects. So actually what turned out was happening is that our lighter was balancing the signal under the bus, and got a little bit of a sparse return from the movement of the person's feet under the bus. And that was enough for the Waymo AI to not only detect that there's a pedestrian there, but also make a prediction about what's going to happen in the future, and keep everyone safe. That's fine, unbelievable, we've got time for one question from the group, if anybody has a key, Jim, sorry, no free codes, not at this one, no, yes, Jim, please. Thank you. Does this work? Yes. I was just saying congratulations on all you've achieved, it's really mind-blowing. If you think about the next five to ten years, really focus on the business model, what are the milestones, what happens in major cities, what's going to be different than where we are today, just kind of walk us through your vision of the future. So we're heads down in execution mode. We've transitioned from intentional sequential de-risking of the driver and keep parts of the business to rapid parallel global commercialization. That means deploying the Waymo driver in more places across the United States, and today we're in 11 cities, operating full autonomous and serving our writers. That we're going to expand those existing places, and we're going to add new geographies and new cities. We're also expanding internationally, announced that this year will plan to offer a service in London and in Tokyo, so you will see us just accelerating that deployment, all in service of our mission. Good news to our team in London. Well, we covered a lot to meet you from the very early days where you could get a lot of distance, with not a lot of technology, to then persisting through extremely hard times in autonomous vehicles, and getting that extra mile. We talked about world models, driver simulator, critic architecture. We got into the hardware, the six-generation hardware, safety, and scaling. But most of all, I hope that we learned a little bit more about Dmitry, the man who's brought the magic that is Waymo to so many of us. And as I've gotten to know you more and more, I'm constantly shocked not only by your brilliance and your persistence and performance, but also your humility, it says a lot for accomplishing this much. Thank you, Dmitry. Please join me in thanking Dmitry for all he does. Thank you. And the many lives say thank you.