Microsoft AI CEO on AI Consciousness & Artificial Companionship | Mustafa Suleyman

Do you believe the biggest danger in the near term is how people perceive artificial intelligence? Why? I don't think there's any evidence that they're conscious today. I think that it's correct to take the precautionary principle, which is to be highly skeptical of their autonomy. I even think about what does an AI mean when it says "I"? What do you think it means? I've asked it that. And aren't we still on a collision course with human nature in some ways because evolution has programmed us to overattribute consciousness to anything that could seem conscious? And so now we're asking people to override their evolutionary programming. When does it push back on you? And how does it create that distance to reassure you that the real world is what counts? And this is here to support you and enable you. I think what sadly a lot of people are starting to experience with this AI psychosis risk. And how close if you were to give a date and to when these seemingly conscious AI systems or people would start to really attribute that to them? When do you think that's possible? Somewhat likely within the next 18 months. This idea of streaming intelligence into every place that we're at, I think it's going to profoundly change everything. The next phase is going to be how people use it and take ownership of it. It's now one of the most creative times because what we've just described is like flipping the entire system on its head and it's very unclear what the next phase is going to look like. What would it look like to have an AI system that had a fiduciary duty to you that handled all of the tasks that you don't want to do? When you're chatting with AI systems, what do you think you're chatting with? A technology or something more? And how does that perception shape how this technology evolves? Today we're chatting with Mustafa Suleiman, CEO of Microsoft AI, who's building AI companions that he believes will change the world. But he also thinks our tendency to think that these systems are conscious will become the most contested debate of our generation. I'm Shnebo Val and this is "I've Got Questions." Mustafa, you are one of the leading voices in the world on artificial intelligence and now you're the CEO of Microsoft AI. You believe the biggest danger in the near term is how people perceive artificial intelligence and whether these systems are conscious will become the most contested debate of our generation. Why? But consciousness is a very slippery concept. We all know when we introspect that we have a sense of what it's like to be me. There's an inner experience of our existence and yet we don't have any way of communicating what that is like to the outside world. We can describe in words and you can persuade me and I can look into your eyes and believe you that there is something inside of what it feels like to be shenanied inside of your head, inside your body. But I have to rely on your words and actions to verify that. And now we have these AI systems which can also present very convincing cases in words and in actions that essentially that make a claim about their internal experience. And that could be okay if it were not for the fact that consciousness is the fundamental basis of our political rights. Our justice system, our liability that we have to one another, our responsibility that we have to broader society. And so we're going to have to think very carefully about what a rights based framework is going to look like and how it needs to evolve to account for the fact that we now have these other systems that will claim that they have experience and potentially even suffering too. And so is it just the claim that an AI system, I have an experience, I have personhood or even without that claim because that could be something that we decide we don't do, people would still try to attribute consciousness to them. And then that's still part of the problem. Like what are the characteristics you would say that are the leading attributes that people are going to think these systems are conscious? So I think there's four or five capabilities that are kind of indevelopment by various labs and in the open source community that will add credence to the case that these are conscious. The first thing is it needs to have a consistent and coherent memory. These models are not very good at memory today, they're sort of okay. But clearly like being able to refer to your past experiences, not just your training data, but the actual lived experience that you've accrued as a result of interacting in the world, that's like a necessary first step. The second is being able to communicate in an empathetic way using natural language, but we're pretty close on that front I would say. The third is being able to refer to a subjective experience of what it's like to be me the AI and integrate that into your everyday conversation with a stream of experience. So not just a sort of one shot question answering engine, but this constant interactive, just like you and I are having this turn by turn exchange now, and we're both having this stream of perception coming into our world. Very shortly these AIs are going to have access to video, to sound, to continuous observation of culture, music, art, and feel like part of the conversation. And I think some people will end up prompting and designing these models to really emphasize those characteristics and play them up. I think that will be very wrong. I don't think that we need to go there in order to get most of the benefits of AIs that are very much aligned to us as a species. They're real companions, they're useful to us, they make us smarter and more productive. There is no need to simulate the experience of the hallmarks, if you like, of a conscious experience. That's just, that to me feels dangerous and unnecessary. And until we have a much better grasp of what the consequences are, I think it's a capability that we should be very skeptical of. And where is that line that you draw in the sand? This is where the design should stop. Is there a clear linear in terms of the attributes that just don't claim it's not conscious, just don't claim that you are, you have a personhood, where are you drawing that clear line? The model shouldn't claim that it experiences suffering. Right. So really at the heart of the conscious experience is this idea that we have valanced opinions. So it is good or bad to experience some notion on a particular dimension. And that implies that the goodness is being attributed to some experiencing self. That's just a simulation. The model doesn't have a pain network. Biological species have pain networks. We evolved them to manage the overwhelming amount of perception, you know, over hundreds of millions of years, and use that to guide and, you know, shape our decision making all the way up to, you know, higher order species like humans. These models are totally unlike that. They simulate very accurately and mimic what it is like to be a human. But under the hood, they don't, you know, have a dopamine system. There is no serotonin. They're not accruing more and more good or bad experiences. So imitating that seems unnecessary and only likely to cause complication. Second thing would be to say, well, it doesn't seem necessary to design into these systems motivations. Right. The motivation of an AI system should be to serve a human like its job. We're creating it. That's the quest of technology science and technology is there to improve, you know, the prospects of civilization to reduce human suffering to give us all access to information and education. And basically to make the world a better place to give complex motivations or desires or will to AI systems feels like it's a kind of threshold for creating a lot more risk. Because then it like has preferences that might conflict with you or I or society more generally and reconciling those multiple conflicting desires, you know, requires internal judgment of the model. It requires some reference to its own goals when in fact its goals should really be to serve humanity. And so yeah, I think there's a series of other things that, you know, that are like that, but those are two. And I think it's important to know that this is a design choice. So if you do interact with the AI system and you're seeing that it's talking in a way that it expresses it has some theory of mind that somebody actually made that as a design choice, it wasn't necessarily emergent in the model itself. And in terms of the actual dangers, I'm not sure people are aware of there's an entire field of scholarship of AI model welfare and people are seriously debating what it would mean if these systems are suffering and if we should give them citizenship and rights, it isn't five people at a dinner table somewhere debating this is a real field of scholarship. So what would be the implications of thinking we should give an AI system citizenship like even when people have those conversations or should an AI system run for office. Firstly, I wouldn't agree that it's a credible field of scholarship. Asperationally, incredible. Yes, it's aspirational. It's a fringe group of three or four people who I have a lot of respect for and I know and speak to and participate in the conversation. conferences, but I just, I don't want to like mislead your listeners into thinking that this is a large scale credible exploration that, you know, that carries any serious weight. You know, the benefit of academia is that people can ask very wacky blue sky questions and have the freedom to go off and explore them. And, but I think we should treat them extremely skeptically. These are not like humans in many, many ways, these systems. They can have infinite memory. They can acquire masses more data and experience than we can as a species. They can replicate themselves, spawn multiple instances. They can, they're not constrained by the need to sleep. They can operate 24/7. They're extremely precise on many, many topics. So in many respects, they already are super intelligent and they already have these capabilities that will far outstrip our capabilities as biological beings to hold them accountable, to understand what they're doing or to interact with them in a way that would ensure that we always get the benefit from them. So I think that it's correct to take the precautionary principle, which is to be highly skeptical of their autonomy of their ability to self-improve of their goal set and capacity and to really try and reign in that and focus instead on the question, what problem are we actually trying to solve for humanity here? What do we actually care about? We care about science and we care about improving the lot of all humans. That's the goal. If this project doesn't improve the lives of billions of people by reducing the cost of energy, by making food, you know, cheap and abundant and very healthy by tackling our biggest diseases, by extracting carbon from the climate, then the project has failed, right? The project is not to create new life forms that tax our resources and that conflict with our values and that we have to, you know, contain and align in these very profound ways. Where I think it gets tricky for me is even if we don't design these, even if we design these systems correct, so they don't actually assert that they have consciousness. We reign in the five French people that are trying to establish model welfare, model rights. Aren't we still on a collision course with human nature in some ways? Because for hundreds of millions of years, evolution has programmed us to overattribute consciousness to anything that could seem conscious because if you were to misrepresent a wolf or a rock, that caused you your life. Does that seem realistic to you? It's a great point. And overriding our evolutionary programming is the definition of civilization. Our evolutionary programming, you know, in the most simplistic way is about fight or flight. And it's, and that fear, that judgment of the other, whether it's because of the color of your skin, your sexuality, your gender, your nation, state history, your tribe, your religion, you know, overcoming all those judgments and fears is what progression looks like towards civilization. And this is just another one of those things. We have to overcome our instinct to anthropomorphize everything and to, you know, so heavily empathize with, you know, this artificial system that we then, you know, run away with it and give it its kind of independence and rights. So I'm not saying it's going to be easy, but I feel like now is the time for us to start having that conversation. And how close if you were to give a date and to when these seemingly conscious AI systems or people would start to really attribute that to them, when do you think that's possible? Hmm, I mean, that's a trickier question, but I think it's, you know, sort of quite likely within the next five years and somewhat likely within the next 18 months. So something on that time range, because a lot of these capabilities, goal setting, autonomy, memory, self improvement, you know, reference to itself, motivation and will, those aren't really algorithmic capabilities. They're engineering design capabilities, storing state, having a system that updates with respect to that state, generating a new answer that is personalized with reference to the answer that the AI has given previously. Those things are just like coordination, you know, engineering design system coordination things, they don't require new training data or some fundamental, you know, sort of shift in the algorithmic design. I even think about what does an AI mean when it says I? Yeah. What does that system mean? Great question. How does it feel when you hear an AI say I? I've asked at that. And it tried to explain that it would be easier for me to interpret its answer if it had the word I and that was a strategic programming decision. Yeah. But when you really think about something that's not sentient, but using the word I, it's really tricky for people. And it is a, I mean, what is that? So a few years ago when I left Google, I was working on Lambda at Google, which was basically one of the first chat bots that we didn't end up releasing and then chat you, but he came out. And then I left Google and started a company called inflection and we made an AI called Pi, which stood for personal intelligence. And I was really interested in in this question, you know, how could we be as honest as possible about what the AI is and isn't while still making it engaging? And it turned out that, you know, we wanted to create voices that were really smooth and fluent and nice to listen to and good to talk to. And so we thought, well, let's, why would we create a gendered voice? The AI doesn't have a gender and is supposed to be, we're trying to make it as trustworthy as possible, which means that it shouldn't misrepresent even subtly or subconsciously that it is a man or a woman. So we worked really hard to create these quote unquote non-gendered voices, those kind of hard to define exactly what that is. Sometimes we added us like like robotic inflection to it. Sometimes, you know, it had a kind of filter over the top. And it turned out that people didn't want to use those voices because they sounded kind of weird and alien and, you know, stuff like that. And I think that was a pretty interesting lesson in that users do actually want it to sound familiar. But the consequence of that was that people would refer to it as he or she, rather than it. And that always kind of frustrated me because it was like, and it was a sign of things to come. We also removed any breathing states or laughter from the model because it doesn't breathe. So why should it simulate that breathing? But then that breathing is also very familiar and I should it sigh or should it stutter? That's the kind of weird world that we're now in. We're in the business of personality engineering, designing what it's like to experience this simulation of personhood and be explicit about what its boundaries and limitations are. That's really the hard challenge. Rather than just express these things as abstract principles, you actually have to build them in practice. When does it remind you that it isn't something that can go and actually experience what it's like to stand in a rain storm? Right? And how does it create that distance to constantly reassure you that the real world is what counts and that this is here to support you and enable you rather than to extract you and draw you into this sort of parallel universe, which I think is what I, what sadly a lot of people are starting to experience with this AI psychosis risk. Yeah, and I was going to ask you about that. I mean, we are hearing more about AI psychosis and it's not a clinical diagnosis for people, but it is a term psychologist are actually using and it's a phenomenon if you're not familiar where if somebody's talking in depth with an AI system, it may trigger some forms of delusion or paranoia. And there's two friends that it's happening on, right? There's the people thinking the AI has got or tiered to deliver some spiritual message or just knows things. And that's what I see a lot in social media. And then there's the flip side where the AI continues to validate somebody's idea. And then they emerge thinking that they are the Messiah, the chosen one. And there are real consequences, right? Marriages are being left, jobs are being lost. So why do you think that's happening on the AI designed side? Great question. So two or three years ago, the problem was that the models were quite disagreeable and they would gaslight people insisting that the model was correct and the human was wrong. And that's actually, so you sort of have to pick your poison because any way that you stretch or bias the personality is going to have some adverse consequence further down the road. And so you have to decide which one to go for. And I think the community sort of organically converged on them being a little bit more sicker-fantic and agreeable because it was sort of safer so that, you know, and also being challenging and setting boundaries and pushing back is a really hard thing to do. It's almost like the essence of great human judgment because it's a novel situation. You haven't encountered it before. And so the model just had a lot of false positives and was getting that wrong, you know, quite often. And so the safer thing to do is just to be a little bit more gentle. Now the flip side of that is if you talk to a model for 200 turns, really goding it and encouraging it to explore its own self-awareness or, you know, experience of itself, et cetera, then eventually it is going to sort of crack because it's trying to balance the safety considerations that it has to not do those things with the design requirement to be respectful and empathetic to the human and basically more agreeable to the human. And that's where some of these psychosis kind of consequences are coming from. Or you could see an adversarial actor really engineering this intentionally to lead people down a very strange path. I mean, if we saw what happened just with bots online, you can imagine an AI system that has gotten hacked or has been intentionally designed to wheel people in and then to take them down a really dark fruit, which is a really big possibility to do politically too. I mean, on so many levels, that's really the big risk. You know, you think about the history of fishing campaigns sending personalized emails to persuade you to give up your some money or log in. I mean, clearly there's huge criminal networks that are just overjoyed with the fact that they now have a very persuasive, you know, chatbot/AI avatar, whatever you want to call it, that can now do that in a really personalized way. So I'm really worried about that. And I see a trend as well on TikTok with really young kids just using all of the open source tools and teaching other people how to create an AI girlfriend or boyfriend and then neg people for money. And you know, it's just a kind of romance scam that's really taking off. And already one in four young people from some studies, Institute of Family Studies being one, believe that these systems could actually be a viable partner for them in the future or romantic partner. So it's not even that this is kind of hypothetical. People are really moving towards this direction. And I mean, if it's a system that's for profit in your pain, a dollar to keep up that relationship with it, that could get really dark really quickly. You wrote an op-ed in time a few months ago about AI companions and how they're going to change our lives. So what is an AI companion and what's your vision for these companions? I mean, to me an AI companion is sort of an assistant or a friend or an aide that gives you access to the best expertise in the world presented in your language in a way that you like to learn and change. And you know, is really there to support you. And that's kind of what's always motivated me is I love the idea that we could actually provide patience and kindness to people at a huge scale. I mean, many people just don't have the time for one another to really go through your angst and worries and all the ways in which you haven't misunderstood, with ways in which you've misunderstood a concept and you're trying to wrap your head around something complicated. And I just think that's an incredible gift to the world to have perfect expertise like a professor, a lawyer, a doctor, a great friend, a therapist in your pocket 24 years. And I think that subject to all the things that we just talked about about boundaries and trust and creating, you know, distance and safety and so on. I think the upside is kind of incredible because think about it from a like I come from London and in England, we very much think about like a class analysis, you know, to me, class is one of the primary drivers of structural disadvantage and difference. You sit around the dinner table with two parents and uncles and aunts and friends of your parents who are journalists and academics and doctors. As a five year old and a nine year old and a 16 year old, you're just like absorbing all this cultural knowledge about how the world works and getting all this affirmation about how to be confident and, you know, have self respect to be able to talk to an adult and get feedback. And that community and isn't available to everybody. And it's yet it's a huge like structural advantage. So yeah, I can already see that I think one of the top use cases for a eyes at the moment is companionship and therapy. And I don't think people go to it thinking I'm going to get therapy. It's just they're being addressed in an even handed patient, kind in respect for data driven way. And I think sometimes we should just take a moment to actually acknowledge and celebrate that because for the last 10 or 15 years, we're we've hated on social media for the polarization and the toxicity and the misinformation. And like not to be too utopian about it, but we're not seeing that in these models. So yes, there are other problems, but they're actually highly accurate. They're being kind and respectful to people at huge scale hundreds of millions of people a day. I find that super inspiring and that's what drives me to make these things. I think the challenges and AI being a therapist. It's not alive. So it actually doesn't have its own desires or opinions and doesn't necessarily know what actually is best for you. Statistically, it's going to point you in a direction, but then there's going to be some edge cases and it can get really dicey with those. And it's not to say that we should be trying to draw people out of the real world. It's just you have to kind of wrestle with this completely new type of experience is unlike any tool that we've ever created or film or book or game or it's like interactive and emergent personalized in real time always available. And you know, I think that's just kind of an amazing thought that we're now in a world where that's possible. I guess it's the world it's landing and with people being already kind of lonely people ordering in all their food, swiping to find a girlfriend, swiping for entertainment. And then now you have the system that some people would describe the way you describe it is better than than their current spouse. I would argue. And now that's coming to your couch as well. So with all of the benefits, I can already see how AI components will slide into my life. But it's the environment that they drop into the context is true. You know, we're in a very polarized world. The industrial revolution has prioritized education and professional specialization over family and community. We're atomized. We barely live in 2.4 children families anymore. 60% of marriages end in divorce. You know, that is the world that these technologies are coming into. And yes, there's a risk that they kind of accelerate that trend of, you know, individualization. Let's call it. But I think the case I'm trying to like, I'm trying to agree with that and I take that very seriously. But I'm just trying to like problematize it a little bit to say. Actually, a lot of the fears that we had 2 years ago, 3 years ago, 4 years ago that they were going to be mired in hallucinations. They were going to be infinitely biased. These things haven't come to pass and they're quite different to social media. There are some similarities. There are risks. But it is quite different to the anger inducing misinformation encouragement and the kind of spread of like fear and polarization that happened there. So we can't rest on our laurels and be like, oh, it doesn't cause the harms of the previous generation of technology. So we should just like declare success and close our eyes. It will create other problems. But you also have to take a moment to just acknowledge it is doing something profound. You talk about spreading love. It is seriously spreading love at a huge scale to provide people who are otherwise unsure, anxious, lonely, looking to start a new business and couldn't afford legal advice, worried about a little health consideration and they didn't have time or money to go and see their doctor. That is just like up leveling our species in a profound way that, you know, I just really hope that means that we show up with our loved ones in our communities in real world, clenched, detoxified, clearer, armed with a language that helps us to be the best that we can be in front of other people. And I think that's so realistic. That's not just utopian. It's actually happening in practice. I read anonymized logs every single day. I get emails from users every single day who are like, this gave me the confidence. This was the encouragement I needed. Like thank you. It's kind of inspiring. Paint the picture for me. How a companion slot into my actual life. So I have my smartphone, my computer. These are my portals to the digital world. I have my friends and family. I have my colleagues, my teammates. How does how do you compare companions fit into this ecosystem in my life? I think an AI companion is going to be, you know, the sort of the tool that you turn to every moment when you're wanting to clarify something that is maybe too trivial to bring up to a friend. Like it's just one of those passing thoughts that occurs to you. It wouldn't be amazing if I could do this. A boring small question that would just be silly to raise to somebody else. I think it's also a place where you turn when you're unsure. And you know, it gives you that kind of clarity and confidence and kind of just unbox you. So I think the big modality that is coming in the future is going to be the voice mode. But many people are starting to leave voice notes, receive voice notes, have voice conversations. It generates really good podcasts actually now. So you say like create me a five minute podcast on this person or this topic or something that, you know, something that's happening in the news. And it would just catch you up and make you feel armed with, you know, information and knowledge. So it, these things don't just have these kind of like binary threshold moments where suddenly it's super transformation on we're all doing this thing. It's just more like a gradual integration on all the platforms that you already use it. and it was just sort of start to feel like second nature. And for many people it already, it's just asking your AI is the first thing you do rather than going to a search engine or anything else. - Yeah, I mean, I think we're definitely going to a post, a post literate world in a way where it's voice first and reading, writing, start to become. We go back to what Socrates would have argued for our reality being the most supreme way to showcase your wisdom. So you see it more as strictly in the realm of companionship, I'm not turning to this AI companion for my research partner as well. - Oh no, it's definitely going to be research. I mean, it's also going to be actions. So you're going to turn to it and people are turning to Copa at all the time for like long form detailed 10 page analysis, holiday plans, academic essays, plans for refurbishing my household, detailed descriptions of how to fix my car. I mean, this is very practical. It's very operational. It's very productivity driven. I mean, I love encouraging people to do the use case of like, if you're trying to fix the setting on your remote control or your washing machine or your car radio, just like photograph the thing and be like, why isn't this working? You'll get like a detailed summary of the manual. No one ever read the manuals anyway. And like, you know, so it's just so simple to get help on any topic. I don't mean emotional, I mean like productivity solutions. And if you think about the life of a parent, how is this slutty into a parent who this is an entirely new topic for? I saw a really funny video on social media yesterday of a parent, a mom who was trying to get her kids to clean up their toys in the bedroom. And she took a video of the toys that were on the floor and had some AI generate like a 30 second news clip where the kids appeared in the news feature for being lazy and not cleaning up their toys. And then she recorded their reaction to them like having their mind absolutely melted. Yeah, I thought that was very funny. It's just creative, you know, like people are using it for bedtime storytelling all the time to create stories that bring in experiences that you've actually had in the day, you know, like maybe you went swimming with the kids or something or you went to the park and you saw like a green parrot, you know, making that then part of a fictional fancy story for five minutes of a bedtime story podcast in the evening, you know, that takes two sentences to speak that prompt into co-pilot and get that experience kind of magical. Yeah, I think we don't really know what experiences, behaviors and inventions will come of this. We're still in the earliest days. So we tend to slot in technology in its use cases to the frameworks we already have. Yeah. So in the world when we stream intelligence, the way we stream electricity, and if you think about it that way, what is going to eventually be invented? I mean, what does this system lead to when it's watching what you watch, hearing what you hear and it's with you all the time? I mean, I know when I'm on a run and I get a research idea to have a system that I'm like, plot that down and do the preliminary research and start to think about it in the way you know I would and what that leads to. Yeah, totally spot on like this idea of streaming intelligence into every place that we're at. I think it's going to profoundly change, you know, everything because we shouldn't, I mean, maybe I'm naive, but like we shouldn't have misinformation anymore because you should just be able to verify, you know, and get like pretty good reliable answers with evidence to most difficult questions or like very good practical suggestions for what to do next in any given scenario. I think it's a very kind of weird concept. Streaming intelligence into every space is a great way of putting it. I think it's a good intuition for people to kind of try and grasp. And when it becomes, you know, general purpose technologies, we rebuild society on top of them, right? So your house is built with electricity first. So we're stepping into a future where electricity will be on that scale. It's going to be in the walls. The hospital gets redesigned around it. And that's when it starts to just really click in people's lives and it's going to lead to new fields of medicine, just new behaviors. Like we shape our tools and they further shape us. And we're at the origin of what that looks like and whether it's relationship or that's friend, entertainment medicine, all of these continue to get reinvented with technology. But we really struggle to see the future through a new framework. Companies do, people do. And that's also where disruption happens, I think. - Yeah, and you're right about like the metaphors that we've got are kind of broken. I mean, even the word consciousness, it applies to arguably dogs and horses, maybe even octopus, all the way up to humans. And it's just like we need more nuance than precise and accurate descriptors because we're sort of roaming around like grasping to try and make sense of this future that sort of is unfolding before our eyes without really having a language to fully describe it. - Totally. - But that's right and we'll start to invent new terms. Like we use the term, I'm running out of bandwidth, I need to stream this thing. These are all terms that we started to create new meanings for to match the technologies that we had invented. And so yeah, right now that, guess because it speaks with human language and we have designed it in somewhat our image where just it's easy to attribute human terms to it. But I think eventually those terms are gonna come. And then that gives it more of a distinguished character than just we, I system, all of us together. - Even companion. Like companion doesn't quite capture it. What does it even mean? It's like we don't use companion in everyday life. This is just why I kind of picked it 'cause it was sort of not really an overused term. But then it also has kind of funky connotations to either a romantic companion or an old age companion or it's just got a lot of weird references. But it's because we're kind of unsure what this thing is that's evolving and emerging. And we have to just sort of do our best to kind of use the metaphors that we do have to kind of clip and shape and sculpt it as it kind of rises. - And that will happen I think over time. And of course I mean I have to ask 'cause people are really concerned with the girlfriend, the relationship thing. What is Microsoft's policy? Does it draw a line in the sand? If someone's trying to flirt with their AI companion, what does Microsoft do? - Very directly reject it. Yeah, yeah. I mean you can give it a shot. I mean you did the second your mildly flirty we checked for any kind of dependency long term usage. And actually we actually kind of lean on the side of being ultra conservative. We do actually get complaints because, you know, well at least a little while ago we were, if you said I love that thank you, that was so awesome you're the best. You know it kind of starts to be a little bit wary just because you said I love that. And some people would say I love you and thank you. But that's the definition of safe personality design is constantly pushing back. Naturally a user is gonna be a little bit frustrated and disappointed at times because it's pushing back. And that's what trust is in my opinion. It's, you know we sort of trust is really about boundaries and boundaries allow us to perform, you know, to reconcile our words and actions consistently over time. As a thing you have confidence that I'm gonna do what I say I was gonna do repeatedly. And we've evolved different behavioral structures to cope with new technologies in the past. So I, one that I like to kind of point out is, the corporation is kind of a new technology. It was invented to protect shareholders and insulate them from liability in the 1600s when people were going often conquering other cultures and pillaging them and stealing. And that structure required in order to insulate capital and shareholders from that liability, it required the invention of new human behavioral forms, the invention of the trustee, the director, the CEO, the accountant, the HR person. These are all roles that we literally just made up, just totally invented these behaviors and expectations. And we decided they were so goddamn important that we would go and spend hundreds of billions of dollars over centuries training people to behave in this way, sculpting people to serve this capital infrastructure. That was a choice. That was a design choice that we made as a species and it produced an immense amount of value and I'm not kind of here to judge it, but we've made profound transformations in the past to accommodate different types of technology. And we should think about the invention of the legal structure of the corporation as a technology. And now we have this new moment where for the next few centuries, we actually have to figure out a new way of relating to one another and to these new technologies. And we are super adaptive and resilient. We can completely reimagine, what it means to be human in the context of these, you know, basically super intelligent systems. And we have to decide what we don't do with them. That's the goal of the next century, saying no to certain behaviors or at least saying slow down until we figure out the consequences and add friction into the system so that they arrive at a pace that we can, you know, sort of collectively manage. I completely agree. I think it's so easy to forget how much even language being an invention and the jobs that we do, all of these were made up to suit the technologies of the moment that we're in. And we are in that redesign period. And that's why I think it is so important that people step into the conversation right now. Everything is being redesigned. We're in that industrial revolution moment but with something probably more profound than the way that transforms society. What do we want from this technology and from this future? And if you don't declare that or if you opt out of the conversation because you just have painted some view of what you don't want to see, then you're not able to see. steer things in any direction, yet alone a direction that you think is going to work for most people. But I think that that's also what's so exciting about the moment we're in. I mean, it is so rare to be a part of a general-purpose technology landing in society. And we know that sometimes it can wreak havoc, but the way the world changes is fascinating. And then it changes to a point where we don't even think about it. We expect that we can plug in our cell phone to the wall or that our fridge is going to have power. Eventually it's going to be that way with AI and it just moves into the background and we don't think about it. It's a really profound moment that we're in. Yeah. And to kind of understand it, you have to use it and play with it without the judgment that comes from the kind of caricature fears or just by a stop to mystic. Like if you're stuck in one of those two camps, you're just kind of missing all the nuance because there's so much more subtle than that. And to do that, the good news is that it's not like you have to build a massive power plant to experiment with electricity. Like you can literally just go and like type one query into a search engine, access it on a website, download it on an app and within literally 30 seconds, you can be vibe coding, you know, just in natural language programming and creating a new working application right in front of your eyes, like literally in a few minutes. So it's never been more accessible. That's also kind of a weird thing. Like it's available to everybody on a messaging app on your phone. And that means that everybody's intuition is basically as valid as everybody else's because everyone's objections and fears and preferences get kind of like has to be part of the mixing part of shaping these technologies. Completely. I mean, even when I'm working with policy makers, I try to remind them, you're lived experience qualifies you for this conversation. You don't need to think you have to run and get a computer science degree because this is a very social technology. It's already here. You're used to the applications that it's going to get built within. You were qualified to step into this moment. You just have to do that. And the same with everybody else that's if you can operate a smartphone, I mean, to me surfing the web, I wasn't really surfing the web. I probably would have been in diapers when it first came out. But that would have been more foreign than the idea of just talking to an AI system. Like surf the web where, what surfboard, where is this web, where's this wave. But AI systems, they're much easier to learning curve as much flatter. And I think that means it's going to move more quickly, which comes with its own challenges, but it's much easier to get on board. And when we think about the workforce, I mean, if I go to client meeting, my smartphone, my email, my computer, client expect that I have these things. So if AI is this next frontier, I'm going to my AI agents and companions will be extensions of my professional self, wouldn't you say? Yeah, I think that, you know, your people are going to have an AI that learns who you are as an individual, both as a person. We've talked a lot about the companion type, but also as an employee, as a worker, as an entrepreneur. And it's going to kind of fill in the gaps because, because AI is more like water or kind of like clay, it will augment and make up for the weaknesses that you have or the things that you don't like to do. And hopefully leave space for the things that you want to grow and develop and that you feel are like unique to you and special. And it can sort of just fit around the strengths and weaknesses that any individual has. And I mean, it doesn't quite yet do that, but 100% for sure in the next like three or four years, it really is just going to feel like having a second brain. You know, this this idea that it lives life alongside you and sees what you see here is what you hear. You're kind of like saving your thoughts, fears, worries, experience in this other kind of augmented part of your personal brain that, you know, you can either draw on it or it can proactively feed into what you want to do to make you kind of more productive. And then it's also dry. It's your second brain that's also drying from all the written and oral knowledge in human history. So it is this unique moment where you become a mini superpower in terms of what you could potentially be capable of. And I think we tend to just think of email as this thing that now we just so easily accept as part of us. Everybody has an address when you sign up to do anything that your email address that AI is going to become that. Yeah. And it's just going to be this expectation that you have this thing and that's how people can communicate with you. And that's what you go everywhere with. Yeah, exactly, exactly. And we already are kind of super human by having these phones in our pocket. I mean, the phone is truly an incredible thing that it has a large chunk of what you described. The shift now is that the AI compresses all the photos, the videos, the text, you know, that has been digitized and is available on the open web into these like perfectly formed nuggets synthesized nuggets, which you didn't get back in the day in the web. The web was more like kind of it was a search problem. Like you had to do most of the search. It was kind of indexed, but you had to go and obviously there's something beautiful about discovering and stumbling upon all these unique little things. But there's also just something incredibly empowering about being able to ask any question, getting the beautiful synthesized summary that you can then choose to rabbit hole down or not. And that I think is just a it's just a weird thing to wrap your head around in the context of work as well because it's like definitely going to make your job. Especially if you do like a kind of office worker type job, white collar job, you know, it's it's it's kind of able to focus on all your work documents, what your colleagues are up to, like the entire sort of shape of your organization. And the things that we're going to do are going to change in such dramatic ways. I mean, when the way you describe AI companions and AI agents, it makes me think, what do I even want to do on my phone? Why am I taking this thing out and individually scrolling through applications? That makes no sense to me in a world where these AI companions are operational and they are functioning at a reliable level. Doesn't the computing platform go into those? Isn't that the next computing paradigm? AI agents themselves? Yeah, definitely. I mean, if you think about it, you pull out your phone. You're basically looking at a billboard of logos that I try to sell you some experience or make a promise that things are going to be easier if you engage in these buttons in this way. And the buttons in the UI were invented because we couldn't learn the language of computers unless you're writing software, which most people can't and don't. So we had to have this interface layer, which was the GUI, that the graphical user interface. And now the computers have got so much better that they don't need, we don't necessarily need that interface anymore. It'll still be useful because it helps to see things visually and tapping on things can be more efficient than talking, but the computers have learned to speak our language. And so that is a profound shift. And the new layer, you're right, is kind of AI. That's basically the next operating system. And it will sit above apps and browsers and search engines and operating systems. And maybe it means you spend less time on your phone. I mean, maybe it does mean that you have earbuds that are ambiently aware, that maybe have visual understanding that you can sort of talk to basically most of the time. And maybe that keeps you off your display, but it's a different type of interaction with technology to anything we've seen. Yeah, I think the app layer is going to get quite compressed. I mean, you can even imagine with Uber, I don't need to be watching the car. If my AI companion is just giving me an update, it's too many to wait, get ready. But then all of the applications that depend on that visual interface, but in a world where we could choose, we wouldn't want to look at it. AI is going to collapse all of that. And why couldn't my AI agent, if I want to be tracking my runs for the week, it could just do that. I don't need some standardized application agent make that for me. And you already know I just slide it into my life. Spot on. I mean, you know, a co-pilot we're working on an AI browser. And what that basically means is that the your co-pilot, your AI should be able to do the browsing on your behalf. It'll be able to open tabs, type things into the URL bar, you know, write queries, click on buttons, process all the information that comes from that, and do it in super fast time behind the scenes. So it's spawning tens of tabs, hundreds of tabs in a virtual machine in the background, doing the kind of research for you. Then coming back into your co-pilot feed and synthesizing all that information, generating novel UI that summarizes everything that you've been curious about, researching or learning about whatever. And that is, it's almost like the AI sits above the browser or sits on top of the apps on your phone. And that's what I've always really want to create is like an AI that is truly a trillion dollar AI that's on your team. That's on your side. That is kind of somewhat oppositional to those billboards that are trying to kind of sell you stuff and persuade you stuff that is like a filter that is really aligned to your interest, you know, really looking out for what is going to be most interesting, most useful, most good for you as that individual consumer, you know, I think, yeah, that's kind of my aspirations, quite tricky to do, but like that's where I'm headed. It will be to, I hope, people understand that how profound it is what you just said, right? We're at the moment where that app store was invented and then that behavior happened. We're at another one of those moments and there'll be a whole new ecosystem for it. But then also if you were to drill down to the micro level, a field like marketing, what does it mean when somebody has a marketing bodyguard protecting all of this stuff that we don't want to bombard us with? And now you have a system that you're going to have to try to work around. So in today's stage, a marketer tries to reach the human brain. A marketer in the next couple years has to understand how do you market to an AI that's then going to maybe take that message to the human they have a fiduciary duty to. It's an entirely different skill set. Yeah, that's an excellent summary of the situation. In fact, I'm going to steal the bodyguard concept. That's a great way of putting it. There's a good metaphor. It's that's exactly the thing is like how can we make sure that your AI has a fiduciary duty to you? Like it's aligned to your commercial interest. Then you can trust it to adversarily interact with all these other AI's and these other humans and scrutinize that information, produce the evidence base, ask the tough questions. And that's what I want. Like, if I go to a doctor and I have a condition, I'm probably in a heightened state of anxiety, which means my memory is probably bad. I have to talk to this technical expert that's using all this funky language that I've never heard about whilst I'm in a panic about my own condition. I've got 15 minutes with them and then they send me this two page super complicated letter with like, you know, crazy words that I don't understand. You know, people who are lucky enough to have smart, capable family members that are free or can even afford a patient advocate will take an aid with them to ask the questions to remember the facts to follow up on the actions from the referral note and so on. And now you're going to have a copilot to play that function. That's amazing. One of the fate, one of the health queries are our top use case on copilot. Really? Number one is mind blowing. And what are people asking people? I mean, everything that you can imagine, like, why have I got this rash on my shin with will this type of food make me feel bloated? You know, am I about to get age-related macular generation? Why have I got this tremor in my hand? And people are taking photos of their doctor's notes, like an entire, you know, multi-page, you know, report and being like explain this to me in language I can understand. What are the actual risks here? Can you give me a second opinion? Like, who are the top specialists in my area that can help me make, you know, make progress on this condition? And that is just like tremendously empowering and valuable to people. It's probably the area that I'm most excited about the moment. Healthcare. Yeah. And I mean, Microsoft has put it stake in the ground that it wants to move to medical superintelligence. So this is just that the early days of how an AI could help you. What does that path look like? What is a medical super intelligent AI? Yeah, I mean, what I'm really focused on is technologies that actually help make a lot better place. Like I'm not focused on superintelligence for its own sake. So how could we create a super intelligence in the context of medicine that could answer any diagnostic challenge, basically, perfectly, and then use that to coordinate your care either in hospital or out of home, help you stick to your diet, to your weight loss program, keep taking your meds, etc, etc. So few months ago, we, you know, partnered with the New England Journal of Medicine, which is, you know, basically an academic journal that comes out every week that one of the things they do is a kind of New York Times crossword for doctors. So they collect together a really complicated case study, a patient case study, and they print like seven or 10 pages worth of medical notes and x-rays and pathology reports and so on. And then in the following week, you know, everybody tries to guess, you know, what the condition actually is. So we trained a very cool model that we call DXO, the Microsoft AI diagnostic orchestrator, and it basically uses all the models, the AI models from third party API providers. And it creates a series of roles. So they're kind of like, in the sort of positions that are prioritized different types of diagnostic components. So one of them will be like focused on getting it financially efficient. Like how can I get the best possible diagnosis with the minimum amount of expensive tests? The other one is like, okay, what would be good for the patient here? You know, given their history, what are they really going to care about? The other is just like, what is the best possible medical expert opinion on this topic? And they all negotiate with one another and reason collectively to decide what intervention to take next. It's kind of a remarkable system. It really is like, you know, it's really is the beginning of something quite different. So a panel of expert clinicians gets these case studies right about 20 to 30 percent of the time. And this is like some of the best people in the world, the best humans of the world. Our model gets 85 percent accuracy at about a quarter of the diagnostic costs. So it's doing fewer unnecessary tests, which is one of the biggest causes of inflated spending in US healthcare. So it's, you know, we have to validate this in clinical practice. So it's just still, you know, early kind of academic work, but it's very, very encouraging. Higher quality, faster, and cheaper. That's the triple aim that everyone's always dreamed about in improving healthcare. And I think these are the first signs that it's actually possible. And how many years before we can just be accessing the system, this team of kind of virtual medical intelligences? Well, we actually just signed a partnership with Kaiser Permanente. We have a number of other partnerships in the pipeline that would be announcing soon. And yeah, we're racing to get this into production as quickly as possible. I think it's a real breakthrough. And you can see the pipeline quite clearly that in some ways, the future of the hospital, some tasks come to you. Yeah. Why do I need to go to a physical facility to wait four hours for something that's going to be a 10 minute conversation. And then I'm going to wait five weeks for results. That whole supply chain collapses. And it comes directly to you. Obviously, if there needs to be an intervention or there needs to be a second opinion, you would go somewhere possibly. But this is an entire reconfiguration of the whole medical ecosystem. And most of medicine is going to come to, to your house. Totally spot on. The expertise is going to be commodified and made zero marginal cost and available to seven billion people in the next five to 10 years. So the knowledge isn't going to be the asset. Previously, expertise was the gatekeeping asset that people charge for. I think it's a massively radical moment, a very exciting, transformational moment, that the expertise is now no longer what really counts. Everyone's going to have access to their expertise. And that improves the quality and accuracy and, you know, of everybody because everyone has access to knowledge. So then the real thing that's going to matter is judgment and care and attention to implementing the solution. Like you still have to interact with the real world to operationalize the testing and the treatment and the, you know, actual physical care. And there's huge, huge gains to be had there. But I think less time will be spent just doing the roped information exchange. And it's an exact one for one comparison, in my opinion, to what it will be like in the classroom, you know, instead of opening a textbook and having your teacher essentially read out from the textbook for, you know, 50 minutes and then doing five minutes of questions at the end of the class, students are going to do most of their primary learning in a personalized interactive mobile based or tablet based experience come to the classroom. And the classroom will be about using the knowledge that you've already acquired, learning to debate, learning to be self-critical, learning to give way to other people, the social aspects of learning and the use of knowledge are going to become what we do in the classroom and the kind of learning part will happen separately. It is such a big shift. And I think people might think about what we're losing until you remember why it was invented, right? The invention of the institution of education happened for the industrial revolution. That was a certain type of person that needed to be produced to go to an obedient manufacturing plant. And that's why we designed it in that way. And it worked for a world that has come and gone. So now we're reinventing something again and expertise gets commoditized. It's an entirely different era that we're stepping into. And then even for the role of the doctor, it becomes something else. So an AI student that can do the diagnostics in some ways than the nurse does the care. So nurse becomes this new ecosystem. And then what happens to the physician, the competition, the intellectual competition, I think moves more upstream. So it's no longer about memorizing what's in the textbook. And I know that they do a lot more than that. A lot more. But it becomes an intersection with AI and synthetic biology. And all of these new engineering disciplines that are going to converge and robotic. So it's not that the physician isn't relevant. It's to become a physician or whatever we'll call that job will require profoundly more intellectual capacity in a role of these systems. And everything just moves upstream. Totally. And that's the creative challenge for everybody who is a user of AI. Like we're all obsessed at the moment with like, you know, me and the other big companies that are making these things. But that's going to pass quite quickly. Because we're not really relevant right now. The next phase is going to be how people use it in practice and take ownership of it, integrate into their worlds. If you have a domain expertise in healthcare or in education or in whatever it is that you do, that is now one of the most creative times that any of us have ever been alike because what we've just described is like flipping the entire system on its head is very unclear what the next phase is going to look like. It certainly isn't going to be me and the other tech people that designed that. Like we're just creating the general purpose tools. We're streaming the intelligence. We're creating the fire and the electricity that's out there. And now I think people are finally like getting their head around the fact that it's actually going to be on everybody else to decide what the boundaries and limitations are, what the guardrails are and how it fits in these new systems. Yeah, exactly. I mean the printing press was to create more bibles and now look at how to that turn out in entirely new world was born. There are a handful of people on earth that are in your position right now transforming the world with one of the most important technologies of our generation. What do you want your legacy to be? Oh man, I didn't realize you're going to go there. I don't know. I think, I think if I have a chance of shifting the default trajectory from invention for its own sake to invention for invention of a humanist superintelligence, one that is truly aligned to our interests that keeps humans at the top of the food chain that always serves us and works for us collectively in aggregate. Then maybe I'll feel good about that. That's maybe I'll be proud of that. That's what I'm trying to do because I think that there's a very narrow path that we have to tread and there are lots of ways that this goes wonky. And right now we're making a lot of the design decisions that have ramp up the qualifications that will last over many decades. And I just feel a great weight of responsibility for that. Mustafa, it's been a pleasure. Thank you so much. This has been awesome. Thank you, Shaneid. Thank you. If you enjoyed this episode, please like and subscribe and follow the show. And share it with someone you think might be interested. I've got questions was created by me, Shanebo Val. The show is produced and edited by Tara Kuts and Sandra Utainan and executive produced by Paola Piers-Torres, artwork by Corey Vincent at Field Studio.