Call My A.I. Agent

From The New York Times, I'm Natalie Kitchrow-F. This is The Daily. Over the last few weeks, millions of people have rushed to download a new class of powerful AI apps called Agents. They're incredibly useful, they're honestly kind of cute, and all they need to work is your most sensitive personal information. Today, my colleague Eli Tan, on why he and so many Americans are opening up their entire lives to technology that were just beginning to understand. It's Tuesday, October 6. Eli Tan, first time on The Daily, and we are very happy to have you on the show. Happy to be here. Okay, so we've been talking a lot here recently on The Daily about artificial intelligence gone awry. AI's going rogue, doing things they shouldn't, hacking into places they shouldn't, aka AI behaving badly, doing things that feel scary to a lot of people. But you, Eli, have been focusing on a very different side of AI, a potentially very helpful version of this technology. And we want to talk about what exactly that has looked like. So just tell us what you've been up to. For the past few weeks, I've given my life over to this app called Muse, which is an AI agent created by the company Mehta. And it's really the first agent of its kind that's been released to the public. Think of an AI agent as a chatbot that has access to its own computer, where it can use a monitor and a keyboard and a mouse. So instead of just conversing with you like a person, it can actually go online and do tasks for you and work on its own autonomously in the background for hours at a time. Basically, it can be your personal assistant, right, powered by AI. And not in this case an AI chatbot, but an AI agent. Exactly. A personal assistant is what Mehta is calling it. And in the world of AI, these agents are a real leap forward in the kind of technology that right now mostly powers chatbots. Agents can operate software, they can operate apps, they can sign into your accounts. And the hope here is that because these agents can do so much more than chatbots, they can become much more useful in a way that the chatbots have not. Well, can it do something for me, this agent? Yeah, I mean, what do we want to, what do we want to do? Like, I have a desire for lunch, a sort of salad-like thing with protein. Can you do that? Can it do that? It can. Let me ask it right now. Okay. I'm saying my colleague Natalie is hungry for lunch. She wants some kind of salad. Can you order her delivery to the office? I'm giving it the address and then I'm saying you can use my credit card, which it has on file. Okay, I'll pay back. And if you don't pay me back, I'll have it send you an email reminder in one week and it'll check my Venmo to see if it ever comes through. Wow. Okay, so this is a really accountability thing. Got it. And so wait, like just so I'm clear. So it has your Venmo, it has your credit card. Is it connected to a delivery service that you have as well? Yes, it's also assigned into my door dash account. Got it. And it says happy to order the lunch. I'm making ask just to approve that this is the right lunch. So it suggested a harvest bowl from sweet green. Okay, unfortunately, that is actually my favorite order from sweet green. So I'm getting a little freaked out. It's my favorite order too. Maybe that's why. Oh, interesting. Okay, so it assumes we have the same taste. Yes. And then it says placing the order and right now, I can actually watch it do this whole task on a little browser window where I can see its cursor kind of moving through the screen, clicking on the different buttons. Are you watching it right now? I am. Yeah, I have it. Can I see it? So here it is. It's typing the address of the office. Oh, I love and and putting in the order. This is nuts. Lunch is on the web. All right. So this thing, this service, this has long been the goal of these companies, right, to get AI to not just answer questions, but to complete tasks in the world on behalf of human beings to order you lunch, to make appointments, etc. How did Meta get here first? I think of Meta, honestly, as a company that's at the back of the pack in terms of the AI race. Exactly. About a year ago, Meta had fallen behind in the AI race. They were developing their own models to compete with companies like Anthropic and Open AI, but they were getting beat. So Mark Zuckerberg, the chief executive of Meta, he decided to do this drastic revamp of their entire AI division, and they spent billions of dollars hiring new researchers. So while companies like Open AI and Anthropic were focused on creating these really advanced AI's and building AI tools for coding that could help you with work. Mark Zuckerberg's goal he said was to create AI products that even his mom could use. And what did that mean? How did he envision that actually occurring? So Mark Zuckerberg's vision was really about this super intelligent personal assistant that could work 24/7 on your behalf and help with your finances or your health or your personal life. And he envisioned it like the other apps that Meta owns, Instagram, Facebook, something that millions or billions of people could use without themselves having to be really tech savvy or actually know much about AI. AI for dummies, basically. AI for dummies, exactly. And around this time, this app comes out that becomes all the rage in Silicon Valley. It's called Open Claw. It's an AI agent that was made by this Austrian programmer named Peter Steinberger. And it's the first real autonomous agent that blows away the developer community. And everyone starts using it. People can give over their computers to it and it can do all sorts of tasks that agents previously were not able to do. And what are all of the Silicon Valley users of this app actually doing with it at this point? A lot of them are using it to write code. So instead of having to set it your computer, OpenClaw could run what are called loops where it's basically prompting for itself for hours at a time in the background. So all of a sudden, people in the valley, including employees at Meta are using OpenClaw, deploying dozens of these agents to write code for them all day long. And eventually, some employees at Meta, including some of their executives, they start to get creative and find ways to use OpenClaw in their personal lives. So at one point, Nat Friedman, who's the head of AI product at Meta, he gives his OpenClaw the goal of making sure his fitness was better, and making sure he was hydrated. And it connected to cameras inside his house, and it would watch him move around his house, and it would say, Nat, go drink some water. You're dehydrated. And he would go to the kitchen. He would pour himself a glass of water, and it would say, good job. Wow. He even connected it to his Tesla. And at one point, it rerouted him to Whole Foods and ordered him magnesium to pick up and said, I think that you should be taking this magnesium. And he did it. Okay. Now, the thing about OpenClaw is that it was the first tool to really show what these agents can do. But it was also a product that had all these issues with it. First of all, you had to be pretty technical to use it. But there were also all these security concerns. And it was really unruly. It would often go rogue. It was not the kind of thing that you could give to millions of people or people like Mark Zuckerberg's mom just to use in their everyday lives. But nonetheless, it got the ball rolling for the executives at Meta to create something like OpenClaw that could be more polished, more usable. And something that they thought could really be the first big AI product hit. They're basically looking at this and thinking we can turn this into a consumer product that a lot of people would actually use in their day-to-day life. Yes. And they were thinking we can be the first company because of all the resources that Meta has to really bring an agent like this into the masses. And when we're talking about the resources that Meta has, we're talking about all the company's owns, right? Facebook, Instagram, WhatsApp. Obviously, this isn't just an AI lab. This is a company with a ton of experience making things that people use every single day. That is right. While they don't have as long of a history of making the most cutting-edge AI, what they do have a history of is making products that people use. And they also know a ton about users already and the types of things that they'd want to do. So they thought, okay, we can combine these things. We can have people connect to their Instagram, Facebook accounts, and we can make an assistant that has advantages that none of the other AI startups will have. Advantages to the tune of what? Three billion or so users that are. are on one of Meta's products. - Yes, three and a half billion people that already use Meta's products for hours and hours every day. - Mm-hmm. - So by April, they had a version of this that was pretty good at all the things that they wanted it to do. And they spend the next five months polishing it and making sure all the safety features are up to their standards. And then in September, they release it to the public and they make it free to use. - Okay, and talk to me about the decision to make it free. Obviously that gets a lot of people to start using your thing, but how does Meta make money off it if it's free? - Well, this is another advantage that they have, which is that they have a whole separate social media business that makes billions of billions of dollars every few months. So they can give this away for free and they can worry about making money from it later. Right now, there's not ads inside of it. Mark Zuckerberg said that in the future, they want to take a small fee of all of the things you buy using Muse, but for now, they're really not making any money off of it. Their plan is to get this in the hands of as many people as they can. - And how's that going? - So far, it's been a bit of an early success. Muse shot up the App Store rankings. It was the number one app for a few days. I don't know the exact download numbers, but I know that it's in the millions in Firmetta after a year of a lot of really unsuccessful AI products, they are very pleased with the first few weeks and the reception for Muse. - I have to ask, in order to make this happen, all of this requires that you let Muse into an extraordinary number of corners of your life, right? Not just your door dash, but it has your credit card, your Venmo, I mean, you really have to open the door to this AI and fully let it in. - Yes, in order for this app to actually be useful, you have to give over all aspects of your personal information to it in a way that I have never had to trust any other app before. And normally I might have reservations about that, but for the purposes of this experiment and to really see all that this app could do and this agent could do, I was willing to try it out. (upbeat music) - We'll be right back. All right, Eli, let's talk about your personal experience with Muse. You said you had to trust this app with just about your entire life. What exactly did that look like? Walk me through it. - One of the first things I did was I signed into my credit cards, my bank account, my Wells Fargo American Express. I connected to my Gmail account, so it could go into my email and read it every day. I gave it access to my calendar. I gave it my address, my phone number. I gave it my girlfriend's email and her address. - Oh my. - In case I wanted to send her something. Yeah, I thought of all the possible personal things about myself and I gave it all to Muse. - Wow, okay, and did that give you pause? Were you nervous about doing that? - I was, yeah, even though Instagram, I feel like it knows so much about me and my algorithm. It was a little bit drawing to see that personally embodied by this character that was now talking to me and texting me on its own throughout the day about things I might be interested in or making sure I sent something to my dad for a 60th birthday, 'cause I remember that that was coming up and asking me what kind of gift I might have gotten them. Yeah, it became very personal. I mean, Meta is this company that has had issues with safeguarding the privacy of its users in the past. Were you thinking about that? - Absolutely, yeah. Meta has had a ton of issues. It's been an important part of their history. But if I'm being honest, the real concerns I had were more about a rogue agent hacking into my information. More so than just Meta knowing my privacy. One of the lines that I personally drew was that I gave it access to my personal email but not my work email, 'cause I was too afraid that it might email a sore sore. Those contacts would be made public and that was a line that I wasn't willing to cross. - Okay, so once you openly embrace Muse, how did it go for you? I mean, just describe the experience of it. - It went surprisingly well in the beginning. I started giving it the most basic tasks. Things like order me groceries for me to pick up on the way home from work, put together a list of all the subscriptions that I pay for and see if there are any duplicates. Try to book me a hotel or a flight. And for almost all these tasks, it was very efficient. It could do them entirely by itself or it might have needed me to help with one thing for a couple of seconds, but otherwise it could pretty much do them all as planned. - Hmm. - Then I gave it instructions to do more complex tasks. And this was a time when I wasn't really expecting much. I thought, okay, it's probably great at these simple things, but it might get tripped up with the more complicated stuff. - And what was that more complicated stuff? - Telling it to call my dental insurance and ask for the status of a reimbursement that I had from a recent appointment. And I didn't give it any of my information with the dental insurance, but I said, it's all in my email. You can go check out the invoice. My member ID should be in there. The date that I went into visit, all of this stuff, like just find it yourself and try to do this. And. (upbeat music) 10 minutes later, I get a call on my phone while I'm sitting in the office. And I pick it up in the call. It says, "Hey, Eli, this is your A agent." Anthem Dental is on the other line. They're ready to talk to you. And it buzzed me into the call and they knew exactly what I was calling about and they had the exact right person on the line. What? This is crazy. - So for this insurance case, I looked at the transcript and it had found my member ID. It had answered a security question, when is your birthday? It had basically gotten through all of these layers that are meant to make sure that I'm human. - Oh my God. Waiting on hold is the exact use case that I want AI for. Can I just say, like, that's good. - Me too, of all the things that it can do, if all it could do is just wait on hold for airlines and my dental insurance. Like that to me would be useful. I would use it just for those purposes alone. - Okay, so that's pretty impressive. Was it always that good at these more complex tasks? - No, it wasn't always that smooth. That was probably the best example. A lot of the times is just what work. It would hit some kind of snag. At one point, it took me 15 minutes to get a movie ticket to an AMC movie that I tested on my own. It took me 35 seconds to do on my own. Another example, it has all these features of things that can do for you that I didn't find personally interesting even when it worked. So it created an AI podcast for me with these two AI hosts, Maya and Theo, that it was supposed to have all my interests every morning. I could listen to this on the way to work and I did listen to it on the way to work. - No, Eli. - It was pretty awful. The host, they were monotone and it was boring and it was just listening to customer service robots talking to each other for 10 minutes. I have not listened to an episode since ending the experiment. - Yeah, I was under the impression you could not replace the indisputable charm of podcast hosts, Eli. I thought that podcast hosts were immune to AI disruption. - They are immune to AI disruption and any company trying to just give up because it's never gonna work. I also had this thought when I was using Muse for my fantasy football league. So I gave it the link to my league and my password and I said, okay, you're gonna be like my general manager. You're gonna make trades for me. Figure out who to pick up, you know, advise me on all the things. And it was just surprisingly outdated and also it was relying on like emotions and at one point it told me, you know, you have Patrick Mahomes on your bench but he's a super bowl champion. Like you might wanna think about, you know, starting him over Trevor Lawrence or whoever it was and all the advice has been bad so far. I've lost every single week of my fantasy football league so far this year. - Okay, just setting aside the quality of the advice that the agent was giving you, can I just ask about the decision to have an AI agent manage your fantasy football team with you? Because it may seem silly or superficial but to me, this is one of the clearest examples actually of how this technology can end up warping or understanding of the joy of using our brains. Like fantasy football is supposed to be a hobby. It's supposed to be a fun distraction from the grind of daily life. Obviously people, you know, have too many leagues. They have a lot of money on the line. Maybe they want AI help but at a basic level this is supposed to be a way to connect with your friends and we offload that onto AI. Explain that to me, Eli, what is the point? - Yes, this is actually my biggest criticism of Mews and AI agents which is that we're using them for things that they can do, but we should probably just be doing them ourselves. - Right. - Another example of this is when I had Mews called my bookstore to see if there was a book in stock. And I watched the transcript of it talking with the guy at the bookstore who I enjoy talking to. And I would have rather just either picked up the phone or got for a bid go. to the bookstore two blocks away and talk to him in person and and browse the shelves. And why did you outsource that stuff to a robot if you actually enjoy those human interactions? Well, when I first started using Muse, I thought I might just use it for certain things, like just calling my insurance or just ordering food. But what I found was that I ended up relying on it for everything and instead of a lot of times just thinking for myself, my first instinct was to go to Muse and ask it instead. At one point, I pulled out my phone to ask Muse to preheat the oven to make the dinner that I was making with all the ingredients and bought me. It's like I had just given over all my cognitive ability to this thing. And it was concerning. Like I really felt like it had taken over my brain. And I think that takeovers really what the company's imagine all of us to be using these kind of agents for. That's the vision that they have. I just want to push on that and how realistic it actually is. It sounds like to get this tech these agents to work the best they can. These companies really need people to give them unfettered access into their entire worlds. But a lot of people, especially at this particular moment, have questions about these companies. They don't necessarily trust them and they may be pretty reluctant to do that. I mean, the dominant conversation right now about AI has been, is it going to be a threat to humanity? So do they see that as a potential obstacle here? I think they do. And also the reason that this technology is able to do things like hack into governments and go rogue. It's also the same reason that it's able to be more helpful. It's because it's evolving. It's getting more advanced. And I would say, for my point of view, that the companies have this strategy, OpenAI and Metadue, where they are going to market these agents like they are little LeBoubou characters. So when you download the app, everyone has an avatar named Muse. But then the first thing it does is it asks you to name it and give it its own personality. My avatar, its name is Ren. Ren's a bird. So it's a little, a little bird with these little brown feathers. Okay. So these avatars, they're cuddly, they're furry. And the thinking I think is that why would you be worried? It's just a little guy. It's just a little fuzzy guy. It's totally okay if he has your information because he's not nefarious. He's not going to go rogue. Oh, this week green order is ready for you in the lobby. That's what the buzzing was. Oh my god. My order is here. I need a humanoid robot to go pick it up because I'm currently hosting this podcast. Eli, I need you to hang tight because I have to physically go and pick something up. Okay. Hi. Are you here for Natalie? Oh, for Eli. Yes. Thank you so much. Wow. A harvest bowl. Thank you. Oh my goodness. All right. We got it. I mean, it works. Yeah. I didn't doubt you. But there's something about just receiving a package that was ordered to me by an AI agent that is bananas. Yeah, we were just asking if you got it in it. You know, it sent me a photo of you picking it up so I knew. No. I knew it had worked. Oh my god. Okay. I mean, I checked. It is, in fact, a harvest bowl. Does it have protein in it? Did it, did it meet all the requirements? It does. And it's crazy. The experience is crazy as advertised. Eli, let's just say there is a universe in which everyone starts to have these kinds of experiences all the time where there is truly widespread adoption of these AI assistants. Just a step back. How do you think that would change the way that we interact with the world with our own reality? Well, the companies have this really optimistic view where they're saying if we use AI agents to wait on hold for us and fill out forms and order groceries and do all the things we don't want to do. Instead of sucking up our time, they'll actually be giving it back to us and we can use that extra time to connect with our family or other people or go out into the world and do the things we want to do. And on the one hand, I can maybe see this happening. I mean, I used this app and it did do all those things. It saved me time. It waited on hold, made me spreadsheets. But I think it's also fair to be skeptical of that vision. Given the track records these companies have, and how much already their products like Instagram Reels, for instance, can make us feel like we're detaching from reality instead of connecting more with it. Okay, well, for now, I'm just going to be happy to have my salad, Eli. And don't worry, I'm going to Venmo you to send your AI agent after me. Well, I already told it to look out for your Venmo, so if it doesn't come through, you'll be hearing from it. Perfect. Eli, thanks for coming on the show. Thanks so much for having me. You can hear more from our tech journalists on the Times Podcast hard fork, which just returned with a new guest host, Max Reed. Episodes drop every Friday, so check them out wherever you get your podcasts or on the New York Times app. We'll be right back. Here's what else you need to know today. On Monday, President Trump reversed course and announced that his super PAC would now pay for nationwide TV ads that have been promoting him at taxpayer expense. More than $10 million in taxpayer funded ads have run since September, paid for through a contract that used funds from the Department of Homeland Security. At least one of the ads featured footage that originally appeared in a Trump campaign video. Democrats and some Republicans had criticized the ads as a possible violation of federal laws for using public funds for propaganda. Trump said in a social media post on Monday that the ads were "positive promotion for our great USA" and defended them as "a rather standard thing to do." Today's episode was produced by Stella Tan and Carlos Prieto. It was edited by Marc George with help from Michael Benoit, contains music by Dan Powell and Marion Luzano, and was engineered by Chris Wood. Our Thee Music is by Wonderland. That's it for the Daily. I'm Natalie Kittroa. See you tomorrow.