Go back

Inside Instacart's AI-Powered Smart Shopping Cart | NVIDIA AI Podcast Ep. 302

39m 55s

Inside Instacart's AI-Powered Smart Shopping Cart | NVIDIA AI Podcast Ep. 302

Instacart’s Chief Connected Stores Officer, David McIntosh, outlines a future where online and in-store shopping merge into a single, seamless, and personalized experience powered by continuous AI learning. The company’s "connected store" vision digitizes physical retail through smart carts equipped with sensors that track weight, location, and shelf contents in real time. These systems combine edge AI processing with cloud analytics to build accurate 3D store maps and understand customer behavior, leading to real-time recommendations and inventory insights. For example, smart carts provide a running total, help users find items, and sync shopping lists across platforms, reducing forgetfulness and boosting sales. The technology also enables proactive stock alerts for retailers and personalized experiences, such as meal planning or targeted discounts. Crucially, the system relies on sensor fusion—combining weight, visual, and location data—to overcome challenges like spotty Wi-Fi and inconsistent shelf layouts. Instacart’s AI foundation leverages over 1.6 billion online grocery orders and real-time in-store data to deliver hyper-personalized experiences. Beyond consumer benefits, the platform improves employee efficiency and retailer operations through predictive analytics and automation. Looking ahead, the company sees potential for agentic AI to automate store-level decisions, such as optimizing product placement, while long-term applications may include robotics and 3D item reconstructions. Ultimately, the goal is a frictionless, unified shopping mode where digital and physical retail converge, driven by continuous learning and real-time data.

Transcription

7617 Words, 42257 Characters

English
Our view is that in five to ten years, customers shouldn't have to think about shopping, install, or online. There will be one single unified mode, powered by this continuously learning AI system that incorporates what customers are doing, install online states of the shelf to build a fully personalized experience. Welcome to the NVIDIA AI podcast. Our guest today is David McIntosh. David is the Chief Connected Stores Officer at Instacart. And we're here to talk about the present and future of grocery shopping and AI and retail. David, welcome to the podcast. Thanks so much for joining us. - Thank you for having me. So maybe we can start with the basics. You can tell us a little bit about yourself, you roll it into a carton, kind of your journey that brought you here. - Yeah, happy to. So fundamentally, I'm a technology entrepreneur, my prior company co-founder, CEO of Tenor, which is an expression search company and an animated drift search company. If you're an animated gift person, probably use the product that is embedded in all the major messengers, keyboard companies, and so forth. When Google bought the company, we had a couple hundred million users, several hundred million queries per day. And then over three years at Google, we grew it to a billion users, over a billion queries a day. And what attracted me to Instacart was that I saw a company that's the leader in delivery online, but had even brought our market opportunity to really digitize the grocery industry, to bring technology to all of our grocery partners. And so in my first year at Instacart, I've led what's called our enterprise business, the Instacart platform and launched that. And you probably know Instacart is an app on your phone, should a marketplace. But what's less well known about the company is we have a very significant enterprise business. So for example, if you go to sprouts.com in the US, that entire experience, website, fulfillment, ads, all powered by Instacart and so as a result, I would talk to retailers very frequently. And what I heard was retailers saying, hey Instacart, you brought me online. You brought my business online, website, e-com, loyalty, et cetera. But I have all these problems in store. I'm dealing with, how do I get more of my customers to sign up for loyalty in store? How should I think about retail media in the store? How do I create a more personalized experience? And then on the other side, we had a lot of customers, a lot of users saying Instacart, I love the convenience and delivery of the online experience. But I also like going to the store. I mean, I'm a channel customer. How can you take what I love about the online experience, the convenience, the personalization, and we're getting to the store. And so the birth of connected store really sat at the intersection of those two insights. And so really our vision for the store is to digitize it and end so that it becomes unified. Our view is that in five to 10 years, customers shouldn't have to think about shopping, in store, or online. It will be one single unified mode, powered by this continuously learning AI system that incorporates what customers are doing, in store, online, states of the shelf to build a fully personalized experience. >> So you teased at this a little bit, but maybe we can kind of double click down. What does the term connected store mean at Instacart? And kind of along with that, maybe you can talk a little bit about how AI is being used. You know, as you say at Instacart, not just the grocery shopping app, but kind of across the enterprise services that you offer. >> You can think about connected store as really digitizing the offline store today. So much of a grocery store today is run very similar to the way it was run decades ago. If you think about the checkout experience, it hasn't fundamentally changed or evolved all that much. But technology has fundamentally shifted in that period of time. And so it's everything from our AI powered smart cards, keep records, which I'll talk about in a minute, to digitizing the deli, putting a screen behind the deli so that it's both easier for associates at the store to prepare an order, but also easier for customers to order. It's connecting to electronic shelf labels so that it's easier for e-commerce shoppers, people picking orders to light up the shelf tag to more easily find, which is about giving users an in-store mode so they can plan their trip to the store, create a shopping list, and then sync that with technologies in the store. It's inventory technologies that build an understanding of the shelf to prevent auto stock so the consumers when they come to the store could get exactly what they want. So you can think about it as really digitizing all the components of the store, and connecting them together. So for example, you can order from the deli, from the Capricart. Capricart connects to our offering called food store, which digitizes the deli. A shop or going into the store can activate those shelf tags, but you could also imagine the Capricart activating the shelf tag to help you find something more easily in the store. The shopping list that you build online, you can then sync with the Capricart in the store and it reminds you, in fact, what to get in the store. So you don't miss anything and realize it when you're all the way home. So all these technologies are connected together. And at the very center of that ecosystem is Capricart, which is a $350 million acquisition I'd let a number of years ago. And the Capricart, you can think about as a set of sensors and a cart connected with an Nvidia Jetset board in every single cart. So the cart has a weights and measures, certified scale. You need weights and measures certification in the US to do produce weighing. It's got camera sensors, multiple camera sensors that not only look at the basket, but also face the shelf. So you can understand what's on the shelf to talk about more. You've got location sensors on the cart. So you can understand where the cart is in the store. It's both a slam approach, but also leverages visuals of what's on the shelf. And then it's a sensor fusion system. So you can imagine in a grocery environment, one of the big problems is that often wifi is spotty. It drops in and out. And often, by the way, there isn't good sell, sell perception if you're going through these stores. - Yeah, a lot of the stores once you get in, yeah, at least I've found the sell drops out. - It's a big problem. And when you think about it, customers want an experience that responds immediately, right? So if you think about the way people use the cart, they're putting items in the cart and then it creates this running total. So what people love about the cart is the fact that they can keep track of their spend, right? They don't have to go to the checkout line and then put things back because they miscalculated how much they're going to spend, right? - I'm laughing because I'm thinking, as you're describing something into one of your, one of Instacards videos or ads that I saw where, you know, it's mom and daughter shopping and mom puts something in and it rings up and daughter puts something in and mom grabs it and takes it out. And you know, it shows how the cart automatically deducts it from your total, which is amazing. And then daughter puts it back in. And mom's like, yeah, okay, just right. But yeah, so maybe we can kind of walk through what happens kind of behind the scenes when I'm shopping. And I want to set this up because I think this part just sounds so cool. It's these little simple things, right? But they really elevate the experience and as a consumer when you get a lot of little things together, you're like, oh, this is a great experience. But the idea that I can be shopping, I can have a bunch of stuff in my paper cart is keeping track of everything. And I can put something in a piece of produce that needs to be weighed. And so if I already have stuff in the cart and I grab an apple and I put it in the cart, what happens? - Yeah, great question. So the entire basket is a scale. And so when you put an item into the cart, it can understand the weight because it already knows the cumulative weight of everything in the cart. But as you're alluding to, it's not that simple because you can imagine that the grocery environment is very complex, right? And so Wi-Fi, as we talked about, is one of the issues. But look, in these stores, there's a variety of lighting conditions. There's tens of thousands of skews. The skews, by the way, change store to store. So even if it's the same banner, right? Even if it's the same retail store, it's going to be very different catalogs or different items. The items change, right? Seasonal differences and so forth. There's very subtle difference in sizes of these items. And then the way that people shop is different, right? So going back to the question on the scale, some people are leaning on the side of the cart, right? Different arms are going in and out. It could be going over bumps in the store. And so it's a very complicated problem. And the expectation from a consumer standpoint is that the responsiveness has to be in hundreds of milliseconds, right? And so if you think about a lot of the AI systems that exist in the cloud, the response I might be seconds, right? And so KetoR approach is doing a lot of these calculations around what's in the basket and recommendations at the edge. And so the way it works is that we have several sensors coming together. We have camera sensors. We have the weight sensor. We have a location sensor. And then we have that sensor fusion system. So we have an edge encoder that processes a lot of those signals at the edge. We also, for longer sessions, longer analysis, we will look at those sessions in the cloud with a separate decoder. And then we'll put the two together what we call an overall shopping decoder, that shopping experience decoder, that builds the best understanding of the customer's basket. And you could imagine that in this environment, there's thousands and thousands of edge cases that can emerge in terms of the way that people are shopping with this product. And so we've really found that it's important to have multiple signals all coming together. In fact, we published a blog post on this, announced it at GTC and actually walked through this in a GTC talk. If you look at just camera alone, there's all kinds of ways that the camera can easily be tricked. And I don't use the word trick to imply necessarily an intent on the user. It's just in the natural way that people are shopping, pulling things out of different speeds. The, you know, a camera getting blocked, the cart getting full all those things we found. You absolutely need those multiple sensors coming together. The weight of the cart, that basket is almost sort of like an X-ray that builds that ground truth understanding of what's actually happening with this thing with all of the camera inputs informing it. So that's the basket understanding side. There's also then an understanding of the shelf. So going back to what do customers love about the product? You mentioned you like going grocery stores. One of my favorite things to do is I'll go to grocery stores where Capers lie. We're now live 100 cities tripled year over year. So in the US globally, primarily US, but we're also live internationally as well. We have coals in Australia and Morrison's recently announced that they're bringing Capers to the UK. So we have an international presence, but primarily, you know, we launched in the US. Sure. And I'll go to these stores. And this one, this one in particular is a wake for wake for and is a co-op on the East Coast. They have shop right, fairway. We're live in about 20% of all of, all of, of, of wake for in stores and growing quickly there. And so I actually will bring typically my wife with me if she's in town because I'll go up to customers and just ask them about how they use the experience and I'll tell you, I get a lot better responses when I'm shopping with someone versus 50 people sort of look and be like, what are you doing? But I'll, you know, of course, they won't know I'm with Instacart, but I'll ask them, you know, why are you using this product? How did you hear about it? What's the value you're getting out of it, right? And when, when I talk to people, I hear number one, it's the running tool. Number one is that I can keep track of what I'm spending. Second, I hear the deals, the discounts, the recommendations on the cart. Three, I hear convenience. You can bag as you go, you can actually take the cart all the way to the cart. The carts can get rained on, they can get snowed on, they're highly durable. In fact, they often will sit outside charging, right? So customers can just pull them, they're highly modularly integrated into existing store operations. And as a customer, you're, you're running total is just checked out when you leave the store or hit the button or whatever. That's absolutely right. You can check out either directly on the cart. Some of our retailers have payment terminals directly on the cart. So just tap your, your credit card or maybe you have a wallet, you know, Apple Pay and so forth, you can be directly on the cart. And then in other cases, you can check out by transferring it to a existing payment terminal. So for example, if you want to pay with cash or you want to pay with a variety of alternative payment methods, sometimes that can be easier to do to do that, that transfer. And that speaks to the modularity of our approach, which is that some retailers would prefer to funnel users through existing systems and exits. They have some want to, you know, want to brand a brand new way, right? But going back to the point around the, you know, how it works, right? The end to end environment recommendations. So how do recommendations work? Well, what we found is that most retailers don't have an accurate plan a gram of the store. What does a plan a gram is a layout of where all the things are at? And keep in mind, even within a given retailer banner, the layout varies often store to store, right? And the problem is that you might know exactly where the cart is to feeding a particular aisle. But if the plan a gram says that there's cereal next to you, but in fact, the cereal is two miles over, right? The recommendation is going to be wrong. And not only is it going to be wrong in that moment, but then users will start to look at the screen less and less over time, right? Yeah. And so the problem with that is that then the utility of the product gets slowly eroded because the recommendation starts to grow to utility. And so the way we've solved that problem is with the combination of AI and Nvidia jets. So we're actually using the side facing cameras. One is a signal to inform the location system, because slam is good. Slam locations are good, systems are good. But in an environment where the aisles are close together, there might be some ambiguity on the aisle two, or the aisle three, and obviously it makes a huge difference on the recommendation. So one signal with the sensor fusion system is, OK, where is the card in the store? The second is what's actually on the shelf? So the side facing cameras are building an understanding of what's actually on the shelf so that it can inform the recommendation system. And then the recommendation system connects with cloud systems. So over the last decade, we have 1.6 billion plus lifetime grocery orders on the online side. So we've been able to make the recommendation algorithm's online very, very good. Yeah, I can imagine. So we're taking a lot of that same technology, that same approach, and we're now bringing it into the store. And the results are extremely exciting. So many of our retailers have shared that they're seeing double digit sales lift from from K from the car. Right. People are spending double digit percentage more. And then we've even been able to more recently add on top of that with recommendation features that further bring online and store signals together. So for example, you're about to check out, you're heading towards the check out. Like there's a screen that pops up says, did you forget? And it surfaces the yogurt that you normally buy, but you just forgot to buy this trip. That feature alone, that one feature alone, drove 1% absolute increase in sales lift. And so that's sick, right? Nearly 1% and that's just to kind of make sure that's a personalized recommendation, right? That's not a like, oh, you're in aisle four where the chips are and chips are on sale on this store today. This is a like, no, I have to change it to be honest, no, you eat a lot of ice cream. So before you leave, did you get the ice cream and that's my, yeah, that's exactly right, right? And so, you know, I think we're just scratching the surface of the new types of experiences, the win-win-wins we can create by understanding location, by having the customer engage with the screen, by having the recommendations be relevant to them. You know, it's good for retailers, it's good for CPGs, it's good for users because a lot of users complain, hey, I got all the way home, I forgot the one thing I came to the store for, it's really, really annoying. You know, that is an example, when you've not dinner up, but when you've been ticking off some of these features and talking about, like, you know, your grocery list and having everything, I'm like, you're peering into my shopping brain and my, like, four different apps with fragmented grocery lists and I forget stuff and everything. And, you know, even just the idea of I'm in the store and I don't know where the ketchup is in this particular, you know, medium access from the cart, like, that's absolutely right. Yeah, I mean, it's I think all these little pain points that people have, increasingly because so many people are shopping online, right, they're saying, look, I sort of expect a lot of the things that I do online to be in store, right? You know, another example of this is we shipped a new recommendation algorithm, or we, we saw a, another one percent plus absolute improvement in sales lift in the store, right? That's a whole whole set on, on top of a lot of it before, right? And what that was doing was better incorporating a lot of the online signals from online delivery into into into the store, right? Now, when you take a step back and you say, you know, go back to your question, how did that all work? This system would not work without the sensor fusion, physical AI system running on the Jetson board at the edge. Yeah, it would not work with the understanding of the shelf, the side facing cameras and that routing through the video Jetson board and both the edge AI system and Cloud AI systems, we have, right? And then that all that would not work with our ability to take signals about what are users doing online, the history we have in depth in building recommendation algorithms online for the last decade into the store. And so, the, the, the magic of the product really sits at all of those three things coming together to deliver that value to users. And I think in general, we're just scratching the surface of the experiences we can deliver. Like we've got gamification capabilities, for example, where customers can get additional deals and discounts and, you know, imagine for a CPG or retailer, they might say, look, this is a customer who shops, you know, once every two weeks. If I give them $2 off their next order, if they shop within a week, it's probably worth it for them, right? And so those are the types of win-win-wins you can construct with an understanding of the user, a screen in front of them that they're highly engaged with, the understanding of the basket, the AI system that we're deploying, you know, at the edge, really scratching the surface of the new things we can deliver. You've talked in depth about the benefits for the shopper as well as the retailer. How are the store employees taking to the new systems and are there benefits of, you know, the card and all of the data flywheel stuff you have going on as you described? How does that trickle down to improving employee experience? Yeah, absolutely a couple of different ways. One thing I will say is that we found adoption of the product absolutely relies on employee support. Employees have to be enthusiastic and supportive to help really drive adoption of the store. And so there's some things that sound simple, but are incredibly critical, like stackable charging. So if you think about it, the cards stack into each other exactly like traditional shopping cards, so they can fit modularly into the store. And when you say stack, I'm thinking I push the card, it goes across, the whole parking lot perfectly lines up in the next card. Exactly. And so you don't need to plug in each card to charge. They charge, yeah. And so that's huge because otherwise the staff would be board it. This is work for me. I have to plug in each card individually, right? So that element is extremely key. We now have the ability to deploy the cards outside. So you can have a stack charger of a stack set of cards outside in the cold winter weather. And so stores don't have to do anything differently in terms of their operations. The modularity element is incredibly critical to success. And just stepping back when I talk about the complexity of the physically eye deployment of Caper, it's these types of things I'm talking about. Okay, let's see, you deploy the cards outside. The Wi-Fi signal is probably pretty weak outside. What do you do about poor Wi-Fi at the start of a session when somebody's trying to log in with their phone number to get the loyalty reports? It's like all those types of things really getting the system to work at the frontiers is very, very challenging. I keep when you're talking about these challenges, I keep going back to the mental image of being at the self-check. And somehow I put something on when I wasn't supposed to, and it says, "Move the item," and I remove it, but I didn't do it right or something happens, and you're in that loop. So with the complexities of everything you've described moving around the store and everything. That's right. You know what you describe to that loop that you get into in traditional self-checkout? You've got, typically, an associate that's standing a couple feet away from you that can come over and help. With a smart card, you're in IL-10, there's nobody around you to help. And so it makes it even more important to get the basket accuracy systems right to understand what customers are adding, what they're moving from the card. You know, to maybe a final point on the trust of associates, this is why basket accuracy is so important. Because the weight system acts as sort of an x-ray, so to speak, for the heart context, that works in tandem with the visual signals coming in again, the sensor fusion system. That's really important, because ultimately associates want to make sure that as customers are leaving the store, the total is accurate. And so building that depth of technology along with retailer-facing tools, where they can keep an eye on the carts as they go throughout the store, and they know the system is highly accurate and performant. That's all part of how you make these deployments really successful at scale, even with the chaos of the store environment. Yeah. Another thing I'll share with you that is pretty interesting is we announced a technology called StoreView, where Instacart shoppers that are shopping in order for another customer can actually scan the shelves of the store with their phone, to build an understanding of what's on the shelf. Yeah. Caper, also with the side-facing cameras, as I described, can build an understanding of what's on the shelf. And so that understanding could then feed notifications to the store, to employees about things that are running out of stock. Yeah. So then they don't have to be reactive. They don't have to wait for a customer to tell them where they don't have to go and do these lengthy checks of the store. It comes to them proactively. And so that's another example of how this understanding of the store, of the shelf, can really improve the customer experience, improve it for users, improve it for retailers, improve it for associates. And so that's really the ecosystem that we're building. Whenever we build features, we've got to think about the users, the retailers, the CPGs, the store associates. How does it all come together to deliver a win for each component of the ecosystem? Sure. David, with all the data you're gathering and everything you're able to do with it, you know, as you said, from Instacart's depth of years of serving billion plus orders to building store views, and this kind of thing. One of the great things about AI, right, is that, you know, the more good data you feed in, the more it learns, the better it gets. How do you, like, do you do system updates for the stores? How do you deliver this, you know, these continual improvements without, you know, the store having to close down for an evening, for sort of inventory type things? That's a quick question. And by the way, I think that's one of the benefits of digitizing the store. As you digitize the store, you make it measurable, which means you can start to optimize it like software, right? And again, going back to an observation, I made it at the beginning. One of the things that was astounding to me when I first saw an Instacart was that most retailers don't really have a good sense of what's on the shelf, right? And, you know, I immediately asked, well, okay, don't don't they have a point of sale, don't they have an inventory system, you know, you can, you can track all the sales and people leave. So what's the problem, right? Not so fast. The problem you have is people are pulling things off the shelf and putting them in their basket. So the item might have been available a minute ago, but then it isn't available the next minute. You have what's called DSD vendors. The CPG's off will have their own employees. And you see that big truck next to a store. That employee will be willing things on the shelf, totally independently from the employees in the store. So it's an inherently chaotic environment. They're always the ones I ask where something is, you know, and they're always super nice, but it's like, I don't actually care. Yeah, it's exactly that, right? And so what we're doing is we're building the best understanding of the store, right? You think about this paper cart with the slam location system, the side facing cameras. We actually had a slide in our GTC presentation showing the 3D map of the store that we're constructing, right? So think about all the things you can build on top of that. You know, one area we're going is we've got a suite called AI solutions. Okay. So what we're doing is we are bringing AI to our retailer partners, things like assistance on their website. And so for example, both Kroger and Sprouts recently announced that they're going to launch Instagram powered cart assistant on their websites, right? But also analytics tools. So for example, you could imagine a world in which we understand that something is missing from the shelf and an AI agent in the background kicks off and starts to communicate with the merchant, right? And say, hey, this is the fifth time this week, this thing has been out at 3 p.m. But the delivery comes at 4 p.m. How could we either make the delivery come earlier? Or maybe we need to adjust the quantity, right? And so you can imagine that once you make the store measurable, right? Once it's observable, you can start to optimize it. And you know, even take the the Capricarts as an example, it's a software system. It's sort of like getting an update to an app, right? Or your phone. We ship regular updates to our retailer partners. So the software is getting better all the time. And then this is probably a little bit, you know, in the weeds, but from a technical perspective, some of these models about the store might refresh us for at least 15 minutes, right? And so they're really designed to get a as close as we can up to the minute view of what's happening. Because again, going back to the example of Autostock, the item might have been on the shelf 15 minutes ago, but then when someone who comes to the shelf either to grab it and put it in their own cart or to pick the order for a customer online, it may not it may not be there. And so that that real time understanding of the store that we're building from the more than half a million Instagram shoppers that go into a store every day. And from these Capricarts that go through the store and are continuously scanning the shelf with the side face of cameras, that's unlocking a foundation for us to build new agentic experiences on top that start to automate the optimization of the store. So then ultimately deliver a better experience to users that makes the associates shop easier and ultimately drives value for the retailer. Yeah, you said you said the A-word agentic. What's your approach to using agentic AI right now? You mentioned, you know, the one agent kicking off and doing singing the background, but is Instacart? Are you deploying agentic systems as part of what you're doing in stores? Are you looking at agents for sort of different expertise kind of, you know, tasks and lines of thinking? What's your philosophy on using agents? Yeah, I think our vision is, I think consistent with Nvidia's vision, which is we believe that there is going to be experts for different tasks, right? And so the way that we're approaching it is that we're building the foundation model for grocery. And so that foundation model is taking in a couple different sources. It's taking in the 1.6 billion lifetime plus online grocery delivery orders, the two billion item catalog that we have. And then it's taking all the in store data. It's taking in this understanding of not just the shelves, but also the click stream. Where is a user pausing on the store? Going back to your example, a kid taking something out of the cart. What convinces you to take something out of your cart? What convinces you to put that thing in? Where are you in the store when you're adding something that you normally don't add to the store, right? And so it's that triangulation of where you are, what you're doing, building the best understanding of what is the most personalized experience for you in store and online that we're then feeding into this grocery foundational model that's building the best understanding of users and the store, which you then can stack on top of agentic applications, whether it's a cart assistant that lets you plan your trip to the store and then take that plan and sync it to the cart when you're in the store or shop it within store mode from an app, right? Or it's building associate facing tools like the example I described, or even CPG facing tools. So another example, if you look at a lot of the shop right wake for instance where in you will find bread from a particular vendor, be in let's say half a dozen places throughout the store, okay? That bread company wants to understand where in the store is my bread being bought, where are people actually taking it off the shelf? And so today what we can do is give that CPG and understanding of stack rank lists, here's the top six places people buy that bread on the store, right? And so then you can imagine the next step in that process is to use use all the location data, the heat map, the 3D representation of the store, and expert agents on top to say, you know what, a CPG, a bread CPG? You should put a seventh location, and here's where you should put it in the store. You should put an eighth location, by the way, and it'll wait out against the cost of a person not putting those locations in other store, because they're burdened by those additional two locations, where it might say, look, you know, in this store, you only need four. Take away those two locations and put that in another store, and so, again, those are all the types of things that can start to be unlocked as you build that really rich understanding of not only the experience in store, but then online and really marry the two data sources together. That's the continuously learning, right, system, right, right, right. What's something that's maybe an insight, you know, that's been uncovered through, I mean, through all of this work, but using AI with, with grocers, with retailers and insight that service that maybe surprised you. Yeah. Well, I'd say the first thing that was surprising to me is often the consumer value props that resonate the strongest seem very simple, but are very difficult to execute against. So take the running code as an example, right, it's amazing that the running total is often the number one thing that users cite when they use the card, right, but to actually deliver that as I stepped through before, how do you build an understanding of the basket and make sure you know what's happening, you remove things, add things, the cart hips, it's bumps throughout the store. Just delivering that is very, very challenging technical problem, but that output itself was very, very simple is extremely, extremely impactful for users. I think the second thing that's been interesting to see is how people, how do behavior changes, depending on these different contexts. So what we've seen, for example, with assistance in the cloud, people are planning a trip to the store using it differently than they would a traditional online grocery experience, a traditional online. So online, this is probably your experience with its cart, right, you're searching, adding, searching, adding, here's the thing I bought before, right, you basically know what you want, you're building a list, you're checking out, versus an assistant online, a user might say, hey, got a family of five. I have this X dollar grocery budget. You know, my oldest child has this allergy, right, second one doesn't like fish. The third one only eats fish. Can you make me a meal plan with that budget for this family, and can you do it for the next two weeks, right? Those are the types of behaviors and queries that you start to unlock with these new agentech workflows. And then by the way, the behavior gets even richer because you then can take what you've done online and bring it in store. And so that cart assistant, I mentioned that Quilgar and Sprouts announced they're rolling out, that will exist both online and the e-commerce website that we're powering for those retailers. And it will exist in cable and the smart car, we should then start to remind you as you go through the store and actually start to shape your behavior, right. And what you do in store will then start to improve the online experience and vice versa. So in general, we look for places where we can really push the envelope on user behavior and add new value to users with these new capabilities, right? It's the same. Another example would be take the bag as you go use case with Caper, often people will put their bags in the cart, reusable bags. And then because they can take the cart directly to their car, they'll unload directly in the car. Those people might have shopped with plastic bags before store provided bags before because of this brand new experience where they don't have to actually take anything out at checkout and put it on the conveyor belt or take it out and re-scan it. Because it goes into their bag once, they then start to shop with reusable bags. So there's a lot of things like that we've seen where because you use AI to make the experience more convenient, more personal for customers, it has second, third or effects in the way that they behave. Yeah. No, that's really interesting. The example of make me a meal plan just makes me think back to like one of the first use cases that I remember seeing kind of anecdotally, I guess, when chat GPT first broke and everybody got on Gen AI was, you know, here's a picture of my fridge, what can I make for dinner? Right. And so that kind of assistance, you know, really resonates, I mean, you're the expert on me, but it resonates with me of that whole like, oh, you know, we had chicken twice this week already. Like, what do I do? Kind of thing. And that can be so helpful in the moment for sure. Yeah. And I think I've been talking more about the user facing benefits as well. One of the things that we launched at GTC, or announced at GTC was that migrating workflows from CPUs to GPUs with respect to ads had really big benefits. We reduced latency significantly. We actually increased click-through rate in the experiments that we land. Oh, no kidding. Right. And so there's, there's, you know, everything here again, we've been talking about very visible to consumers, brand new use cases. But when we think about the impact that AI is having at the edge and at the cloud, it's really across the board, right? It's, we're pushing the frontier with physical AI, with these carts in the store, with shoppers scanning the shop, understand what's happening at the edge, but also by bringing these systems online, there's also impact that you can drive to recommendation systems, ad systems and so forth, right? And we're really seeing those, those two pieces come together. That's what we mean. So we're collecting millions of sensor inputs daily now, that was one of the things we announced at GTC, and then combining with our, with our, our, our online set of data. So David, what's next? Yeah. You know, when I think about it, it's really the vision I laid out for you at the beginning of the call. It's on a 10 year horizon. We think that customers won't have to choose between shopping and store or online, right? There won't be a, a, a, a solid line between the physical store and the online store. It's going to be one single unified mode, what you do online, plug into in store, and vice versa, which is continuous, and it's going to be a continuously improving loop behind the whole system that ultimately makes the shopping experience more personalized, more seamless. So what does it say about me that as you're describing this, I want to like slow down as I drive by the store, hit the button that opens my hatchback, the groceries that get pulled in. Like, you know, you're talking about that kind of, there's no line between online and offline, right? I'm like, yeah. Yeah. I mean, look, if you want to talk even longer horizon, I do think, you know, if you think about the data that we're collecting and the, and the 3D maps, for example, of the store, the slam system, you could easily imagine that that starts to, in, in the, now I'm talking about this in future that could start to unlock robotics workflows, right? Because again, you know location of things in the store where they're at. You even know, by the way, how heavy is each item? Why? Because the card is weight every single item that you put in the card. You've got multiple cameras looking at the item as it goes in. So you could imagine starting to create reconstructions of that item over time, you know, potentially in 3D. Yeah. You could imagine with, with that understanding, it could even start to unlock those, those types of, of use cases, but that, by the way, really goes back to the vision of connected store, which is how do you start to unify the experience for some people? It's going to be delivered. It's going to be pickup for some people. Sometimes they're going to want to go into the store. They're going to grab a card. They might even grab a portion of their order via pickup. It might be a hybrid experience where, you know, as you, you approach the store, you've already ordered, you say, look, I know I want to get, you know, this type of cereal, the second type of cereal, but I want to pick my own produce. So you can even imagine maybe that picked order is actually waiting for you in the card. My card has your name on the front. You grab it from the line and then shop, you produce yourself, right? And so that, that, that totally is consistent with, that, with the future we imagine, which is, again, that we're really breaking down that barrier between in store and online. We're really turning into one single unified personalized mode for customers. It's exciting stuff. It's incredible. David, for listeners who want to learn more about InstaCard, about the CaperCards, about everything you've talked about in the, in the future, best place to go online. InstaCard website is their technical blog, something specific about the cards, social media accounts to follow. Where would you direct them? We've got a lot of this information on the InstaCard.com website, both pages around our enterprise technology. I'm also a blog where we posted the announcement with Nvidia for, for, for GTC. And so for an audience that's looking for more of the technical details, we've got in that blog post a pretty lengthy description of, of what we're doing. And by the way, given the reaction, we saw GTC from the audience, we're planning to do a lot more of that actually in partnership with Nvidia because there was a broad appetite from the audience and really going deeper. But that, that blog post already has a fair amount of detail, I'm really how we're pushing at the frontier of physical AI. So that's, that's one resource. And then I would say for a more consumer heavy audience, we've got videos that you can search on YouTube, both videos we've shot of the card, but also there's been a number of TV broadcasters that have covered, covered Caper and, and done their own segments. Yeah, excellent. David, this has been great. Thank you so much for joining the pod. And you know, as I said, I've always been a fan of the grocery store, I know where it is, I just like going in. So I'm looking forward to getting my hands on the Caper Cart myself. Awesome. Thanks for having us. Appreciate it.

Podcast Summary

Key Points:

  1. Instacart envisions a unified shopping experience in five to ten years, where customers seamlessly transition between online and in-store shopping without needing to choose between them.
  2. The "connected store" initiative digitizes physical retail environments using AI-powered sensors, such as smart carts with weight, camera, and location sensors, to provide real-time, personalized shopping experiences.
  3. By combining edge AI (on carts) with cloud AI and offline data, Instacart builds accurate, dynamic models of store layouts, inventory, and customer behavior, enabling smarter recommendations, automated stock alerts, and improved retail operations.

Summary:

Instacart’s Chief Connected Stores Officer, David McIntosh, outlines a future where online and in-store shopping merge into a single, seamless, and personalized experience powered by continuous AI learning. The company’s "connected store" vision digitizes physical retail through smart carts equipped with sensors that track weight, location, and shelf contents in real time. These systems combine edge AI processing with cloud analytics to build accurate 3D store maps and understand customer behavior, leading to real-time recommendations and inventory insights.

For example, smart carts provide a running total, help users find items, and sync shopping lists across platforms, reducing forgetfulness and boosting sales. The technology also enables proactive stock alerts for retailers and personalized experiences, such as meal planning or targeted discounts. Crucially, the system relies on sensor fusion—combining weight, visual, and location data—to overcome challenges like spotty Wi-Fi and inconsistent shelf layouts.

6 billion online grocery orders and real-time in-store data to deliver hyper-personalized experiences. Beyond consumer benefits, the platform improves employee efficiency and retailer operations through predictive analytics and automation. Looking ahead, the company sees potential for agentic AI to automate store-level decisions, such as optimizing product placement, while long-term applications may include robotics and 3D item reconstructions.

Ultimately, the goal is a frictionless, unified shopping mode where digital and physical retail converge, driven by continuous learning and real-time data.

FAQs

A connected store digitizes the physical grocery store experience by integrating technology like smart carts, electronic shelf labels, and AI to create a seamless, personalized shopping experience that unifies online and in-store activities.

AI powers real-time basket tracking, personalized recommendations, and shelf monitoring using sensors in smart carts, enabling accurate, context-aware suggestions that adapt to individual customer behavior both online and in-store.

Smart carts, equipped with weight sensors, cameras, and location trackers, provide real-time basket tracking, shelf visibility, and weight measurement, allowing customers to manage their spending and find products more easily in-store.

Instacart uses AI combined with sensor fusion—tracking cart location via SLAM and analyzing shelf content via cameras—to create a dynamic, store-specific understanding that accounts for layout differences across locations.

Retailers gain better inventory visibility, proactive stock alerts, improved sales through personalized recommendations, and enhanced customer engagement via real-time data and AI-driven insights.

Instacart continuously collects data from millions of shopping sessions and uses it to train AI models that improve basket tracking, recommendations, and store layouts, enabling a self-learning, adaptive shopping ecosystem.

Chat with AI

Loading...

Pro features

Go deeper with this episode

Unlock creator-grade tools that turn any transcript into show notes and subtitle files.