281 - More Chat, Less Bot - Jeremy Utley, Kian Gohar, Henrik Werdelin

Welcome to the You Are Not So Smart Podcast. Episode 281. When we first started seeing results flow in, my first thought was, oh no, oh no, you know, and actually it's a good sign as an academic researcher when you get the data back and you think, oh no, because then you wait a few beats. And I remember that first day on and on the phone looking at stuff and I go, oh yes. My name is David McCranny. This is the You Are Not So Smart Podcast and that was the voice of Jeremy Utley, a professor at Stanford University specializing in creativity and entrepreneurship. You may remember him from a previous episode of this podcast where we discussed his book Idea Flow. And since we've recorded that episode, I have visited him at Stanford to see the inner workings of the D school where he teaches. Also known as the Hasso Platner Institute of Design, except no one calls it that. They call it the D school. And Jeremy Utley co-founded a very popular program there called Stanford's Masters of Creativity. They literally take students from all walks of life and put them through programs and activities designed to unlock their creative potentials. It's pretty amazing and I visited as research for my next book which is going to explore just what the word genius really means and why that's such a hard question to answer. Scientifically, linguistically, culturally, historically, all the big ALL-Y words, it's going to be a weird book, I think you're going to love it. But yeah, the D school, the D school is one of a handful of places around the world devoted to systematically, methodologically, unlocking people's genius while also teaching them not only how to generate lots of ideas, but how to sort the good ones from the bad ones using the latest tools available. Which now includes AI, chat GPT, which is the subject of this episode, and the reason you just heard Jeremy say, "Oh yes." Yeah, surprising results, that's what he's referring to. His research into how teams perform while using AI to assist them during brainstorming sessions and that research delivered something very unexpected. In case you haven't heard AI tools like chat GPT are here and they are going to change just about every aspect of how we work. So if that's something you're thinking about, if that's something you're worried about, that's something you are interested in, and I think you should be, perhaps all of those things, this is an episode about some of the earliest research into that topic that just got published. Here's Jeremy and his research partner, Keon Gohar. Talking about why they wanted to do the study, we're going to explore in this episode. Both Keon and I, when chat GPT came out, we started playing, and because of the roles we played, advising organizations and building capacity around innovation, we started hearing kind of murmurs in our respective networks around what are the implications for innovation? What are the implications for creativity? Is ideation dead? All these kinds of interesting questions, and the truth is we had hunches ourselves, but we didn't really have anything more than a hunch. We had our own personal kind of call it anecdotal experience, but not much more. For me, the question was, what can we demonstrate apart from anecdotal observation and what kinds of claims can we make about the impact of generative AI on the way teams work when they're trying to solve problems? But I got to actually, I have to pass it to Keon because this whole project was his idea, and I'm dying to know, actually, Keon, I'll put you on this, but what were you thinking when you concocted this crazy research project in your mind? It was a crazy idea. It was the beginning of 2023 January, and so in my work, I helped teams become hyperforming through solving problems, more collaboratively through exercises and practices and behavioral changes that allow them over time to get better at what they do. And when CHAPTGPTV became publicly available, I was really interested in exploring what happens to the world of problem solving and how do teams collaborate together one-on-one? When all of a sudden you have this other tool that can help come up with ideas to problems that your team may have. And so I was really interested on exploring how the impact of generative AI on a team basis, and how do teams use it, how will this improve their problem solving capability, how will it make them feel, how will it make them a higher-performing team when they have access to these kinds of exponential digital tools that they've never had before? So they did the study. And based on the results, Jeremy and Keon have developed a whole paradigm for getting the most out of team-based AI-assisted ideation. It involves Taylor Swift, the Einstein-Lung Effect, GPT-Prompt, Cognitive Biases. We'll get into all of that in just a moment. You'll hear all the details, which have a really interesting You Are Not So Smart angle that I'm eager to tell you about. And you'll get some incredible, useful, science-based, actionable advice. But first, because this is relevant to what you're about to hear, you should know that Jeremy just launched a new podcast called Beyond The Prompt. The premise is that this is a show about how professionals around the world are using AI tools like ChatGPT at their companies right now in their work. It's hosted by Jeremy and Henrik Werdelen, who is an entrepreneur known for starting barkbox, a monthly dog toy and treat box subscription service. He also founded Prehype, a venture-building R&D group in New York City. Their show features conversations with experts about how to leverage AI to accelerate in a word, business. And since Jeremy and his research partner, Keon Gohar, just published a fresh scientific paper detailing their study into how to do just that, I sat down with all three of them Jeremy, Henrik, and Keon to record an episode about some surprising psychology they discovered that we will all have to take into account when using ChatGPT and other AI tools. Here is everyone introducing themselves, starting with Jeremy Adley. I've spent the last 15 years teaching at Stanford's D School, wrote what I thought would be the world's greatest book on brainstorming and idea generation. That was published one month before ChatGPT came out, which I considered to be the equivalent of writing the world's greatest book on retail, prior to the invention of the internet. Which is to say, I have the sands upon which my expertise is built are shifting rapidly. So about a month after ChatGPT came out, I left my operational responsibilities at Stanford to go all in, exploring the implications of AI on innovation and entrepreneurship and organizations. And here is Henrik Wartolin. I'm an entrepreneur. I've been building companies for most of my career and I'm very interested in, I guess, the art and science of how you built from scratch and make it a little bit less stressful to run. And for about AI, I've had access to OBI and a bunch of the other fundamental platforms for a few years and I'm very intrigued in both applied entrepreneurship but also applied AI. So a little bit less of the bits and bots and a little bit more and like how do you actually get humans to use it? And finally, Keon Gohar. Yeah, sure. Hey, gentlemen. Great to meet you all. Keon Gohar, I live in LA. I run a leadership development firm that helps organizations with high performing team behaviors. I wrote a book, came out last year on the future work, best on best practices that the most high performing teams in the pandemic deployed and lessons learned from that. Formerly, I was an executive director at Singularity in Silicon Valley for many years and also an executive director at the X Prize here in LA, designing moonshots to solve big challenges for humanity. Entrepreneurs, business, science, academia. So while recording altogether somewhere along the way, we brainstormed a bit, which you know, this makes this meta already. And we decided that I would make an episode about this study for you or not so smart and they would adapt that episode for their podcast beyond the prompt. Business, innovation, idea, it's all very meta. So let's get into it. AI assisted ideation. The Taylor Swift method, Dynch Dalang Effect. Before we get into just what all of that is and what Jeremy and Keon found in their research, let's take a moment to let them explain what exactly is ideation. Ideation, I think, is developing effective possible solutions to a problem and teams can define that however they want, but coming together virtually or in person or in the metaverse to figure out what are the different possible ways a problem can be solved with different modalities. And in my mind, tell me if I'm wrong, it's coming up with good ideas like you're wrong, stop, stop. No, no, no. Okay. Tell me crazy. Thank you, David, for the opportunity to get on my high horse for a second. There's why good? Why introduce the concept of quality? That's what everybody does, right? Everybody says ideation is coming up with good ideas, but the problem is good and good. I don't think quality has any bearing on the conversation at the beginning and effective ideation. Insofar as ideation is effective problem solving, effective ideation is generating ideas without regard to quality. And one of the key limiting factors for most teams and individuals who are trying to engage in effective problem solving is their fixation on good. I love this. I love it. In this scenario, I mean, doesn't necessarily mean come in with the greatest ideas ever. Let's put our heads together and we come up with the greatest idea that's ever been had about this particular project or problem. And some of the materials that you, in which you described the study, you had these beautiful quotes. I think it's line is paulding. I've got a note here to have a good idea. You need to have a lot of ideas. Keith Simonton. Minza Lifetime Achievement Award winner, he said quantity is the single greatest predictor of quality, beautiful, I'm quoting things that you've sent me, you know, the more willing you are to have a bad idea, the likely idea, have a good one. What about this Taylor Swift thing, the, hey, Kean, what did Taylor Swift have to say about this concept? Oh, Taylor. She is obviously the master of creativity, you know, when she was accepting her award at the I Heart Radio Music Award this year for most innovative artists, she comes up and she says, I want to, I want to make all my fans be aware that for the few amazing song ideas that I've developed, I've also had hundreds and thousands of terrible ideas that allowed me to ultimately get to the few dozen good ideas that I've had that have gotten me on the stage. So, you know, the master of creativity says to her fans, don't be afraid of having lots of bad ideas because of bad ideas are just cousins of having a better idea on your next iteration. Okay, so that's ideation and this is something people can do in groups and this kind of group brainstorming works better when bad ideas are totally welcome. In fact, permission to have bad ideas means you can work in stages so that sorting the good from the bad can occur down the line. So how would that work when chat GPT enters the mix? When teams are directed to use it as a tool in that process. This was the major question, Jeremy and Ken set out to answer. So to do that, they got lots and lots of people at real companies whose real jobs are to come up with innovative solutions to real problems. They put those people into lots and lots of teams, some using chat GPT to help some not. And then they compared and contrasted the outcomes of their work to collect lots and lots of new, never before quantified evidence based on those results. See, prior to this research, no one had looked into how teams might use chat GPT, not scientifically. The hypothesis, the general assumption was that if you give a group of people, generative AI, it would of course lead to a revolution in corporate innovation. And that would lead to new automated workflows, vastly increased productivity, supercharged brainstorming sessions, that sort of thing. But that is not what happened. Here's how the study went, as I understand it, and you feel it in my gaps here, you're like, hey, here's some actual challenges that businesses are facing. How do we improve customer service in this particular division? How do I develop this new product? I need some new internal training resources. And so then you hand this over to, these are problems that people are looking for, some sort of solution or ideas about how we can work on them. And you hand these to groups that are divided into some are using AI, and some are not using AI to ideate, to brainstorm. So starting there, let's imagine that we're talking about this group's not using AI, this group is using AI, you can back and forth, Jeremy and Ken, like, what did you see in starting with the groups that didn't use it? What kind of things did they do in this situation? And the groups that did use it, what did they do in this situation? Well, importantly, the groups that know of one another's existence, right? So part of the research is you can't let people know the jig, right? So as far as problem solvers are concerned, they've been invited to shop to a problem-solving session. And that's why the problem was so important. It's had to be relevant to the organization, not only because we wanted to study the impact of AI on problem solving in context, in a meaningful context, and not theoretical problems, but real practical problems that businesses are facing. But also because that's what it requires to get human beings engaged with a problem-solving exercise. They go, oh, yeah, our internal training materials do stink. We do need to improve those, or, oh, yeah, we had been wondering whether we want to enter this kind of adjacent market, and how would we do that, right? So it's not only important for our study purposes that the problem be meaningful, it was also important to participants to say, hey, I'm going to spend a couple hours of my life working with other people in the company. I want to feel like this is going to make a difference to the company, right? So as far as they're concerned, they're not a part of a study. As far as they're concerned, they're a part of a problem-solving exercise, right? And so for us, and we scheduled them such that they didn't know of one another's existence, and we didn't tell either one of the others' existence, right? So the non-AI group got a world-class brainstorming activity, right? Yeah, and I happen to be world-class innovation facilitators, right? So they got a premier brainstorming activity, and AI teams got a premier brainstorming activity, and we're also given access to generative AI tools and given a brief primer on how to use them. Right, but so in both cases, importantly, I just wanted to state that neither one knew one that the other group existed or two that we were studying, how does performance differ? Now, as far as both of them are concerned, they've been invited to a problem-solving exercise, it seems relevant to the business, and they're willing to devote a couple of hours of enthusiastic engagement for the sake of improving business outcomes. That's kind of like the starting point. I would also add that what we found was interesting, teams who didn't have AI brainstormed like how you and I think about coming together and developing ideas to a problem. You know, we're in person, and we're virtually, and we whiteboard it out, and we come up with ideas, and we prioritize it. It's just like we've done it for decades. What we found was with teams that had access to AI, they had different approaches to how they used it. Some teams were totally quiet, like they basically had a resting AI face because they were like staring into their computer with chat GPT and trying to get up solutions on their own, and they weren't really talking to each other, even though they were like right there. It was like study hall, it was hysterical, and chat GPT faces a phrase that Keanu and I came up with, because it's like, you watch these people while they're in the session. It's like they're oblivious to one another's existence, it's kind of fascinating. So there was like that, there was that modality, and then there were others who were much more communicative with each other during the process of generating ideas with chat GPT. And so just exploring, how are some teams super quiet and solitary, and that depends on the dynamics and individuals, and who takes the lead and how they structure the exercise versus teams who organically are much more iterative with each other and talk to each other and say, "Oh, this is what ChadGPT said. What do you think about this?" And then they build on top of that. And so just being able to observe the two different kinds of styles was very different than the control group, which we didn't have access to AI, and would do brainstorming like how we've done it for decades. We've got the study, we've got the idea of it, we've got these two different groups, the control group, they pretty much did the same sort of thing that people do, but the AI groups are doing all sorts of different stuff. And this makes me feel like, as I'm reading the study, "Oh, yeah, and the cool thing is, it's such a cool takeaway at the end," which is these people with this incredible tool, making, putting hundreds of ideas together in all these different marvelous ways. And then, "No, I didn't get the outcome, I thought I was going to get. I'm not going to say it for you, because it'll be more interesting to hear you put it out there." What did you discover about the quality of AI assisted groups versus non-AI assisted groups in the domain of ideation, please? You're exactly right, David. When we first started seeing results flow in, my first thought was, "Oh, no. Oh, no." And actually, it's a good sign as an academic researcher when you get the data back and you think, "Oh, no." Because then you wait a few beats. And I remember that first A.K. on the phone looking at stuff, and I go, "Oh, yes." Right? And you realize at that point, we had assumptions about what we were expecting. Theoretically speaking, you read a lot of the research right now. It's like, the theoretical ceiling is very high for AI assisted teams. And so, I'm expecting, it's like, the question was, "How many multiples more ideas are AI assisted teams generating? How much broader is the divergence of possibilities that they're met?" Like, to me, it was a question of what's the multiple number, right? What's the expansion factor? And then to see, AI assisted teams aren't generating more ideas. Often, in fact, they're generating less, substantially less. And furthermore, they aren't generating better ideas. But what was wild is, they weren't generating worse ideas either. They were generating a modest amount of ordinary material. And the flies in the face of all of our expectations, I thought we're going to conduct a research study to talk about how much better AI assisted teams do. How much broader their thinking is. And it just wasn't the case. Now, there are times where, as we've mentioned, AI assisted teams do outperform. But that comes down to a particular orientation towards the technology. And we can dig into that. But the fundamental finding, it was one of those, "Oh, no, moments that really helped us spark, I think, a pretty, interesting insight." Yeah, there were moments when Jerry was like, "Yeah, this is crazy. We're doing this wrong." No, let's just wait to see what the results come out. And as we saw more and more of the teams developed the same kind of results, it became obvious to us that, yes, there are limitations in how most humans use AI to get more ideas and better quality ideas. But the teams that did really, really well, they did it differently. And so, this became very clear to us that this is not just a technology limitation, rather, it is actually a limitation of how humans expect the technology to help us solve the problem. And that requires us to change our behaviors, our biases, and how we work going forward with man and machine. I mean, one of the things that's so fascinating about the study is that AI and the way that we interface with AI is such a seductive, easy interface. Just to remind you, that's Henrik Woodland co-hosted Beyond the Prompt. That you kind of assume that you can't really do anything wrong, right? Like, it's a little bit like brainstorm assisted by whiteboards or posted notes. So, you're going like, "How can you not use these in the correct way?" And it's just such a mind-blowing thing that because it's so, it's seemingly so seductively easy, you kind of like don't even realize that you could do it in your own way. So they have these corporate innovation teams, lots of them, several groups, and they sit them down in groups and have them engage in collaborative problem solving. On the sorts of things these sorts of teams usually work on, while at the same time, other groups perform the same tasks in the same way, without the help of Chad GPT. And they measured not only how many ideas these groups produced, but how good were their ideas, how clever and useful were their solutions. And they did that by asking, as they put it, the problem owners, the person or persons whose problem the team was trying to solve. And those people ranked the ideas as A, B, C, or D in quality. And what they found was that the AI-assisted teams tended to under-produce a moderate stack of C-quality ideas. The solutions were mid-okay, not great, and not terrible. Teams using AI converged on average quality solutions. And this is what makes this such a great, you are not so smart kind of study, when asked to rate the ideas. The people giving out the grades, unaware of which had been AI-assisted and which had not, assumed the few great ideas coming their way, must have been the ones produced by the AI-assisted teams when they very much were not. As one greater of ideas in the study said, "I believe the same lie. I thought the AI-assisted group had more and better ideas." So, what accounts for these results? Well, according to Jeremy and Keon, a pre-nicious cognitive bias was at play. The good old, I inched along effect. I can talk about, I can wax poetic about this for a minute, but I'm alone to hear your take before I start talking a little bit about what this even is. I will just contribute, this means setting or attitude. This is something that people do when they have done things kind of sort of like that before. But go farther with this idea if you will, Jeremy and Keon. Well, you actually use saying that they've done things like this before, I think is actually a really helpful addendum. Chat is such a natural interface. It's almost like the deck is stacked against us as humans in collaborating with AI, because it seems like so much other stuff we've ever done. Which is exactly, as you know, David, what Abraham and Edith Luchins found back in 1942. The water jug. The water jug experiment, yeah. Okay, let me drop in here to explain the water jug experiment from the 1940s. Subjects were split into two groups. One saw the series of water jug puzzles that the participants would learn all had pretty much the same solution with many steps. The other group didn't do any puzzles like that beforehand and they served as a control. Then both groups were given a new kind of puzzle that required a different method of problem solving from the kind the groups that had solved problems beforehand had learned how to solve, but you could use that previous method. It just would take forever. Long story short, the control group tended to settle on the simplest possible solution to the problem while the other group tended to settle on the more complex, laborious solutions that required more steps because that was the sort of problem solving they had learned from the previous puzzles. They had formed a cognitive bias and they were biased in favor of using familiar methods rather than seeking out innovative approaches. Oh, and if you'd like to try to solve one of those puzzles, here's an example. You start with three jars. One can hold eight gallons, another five gallons, and another three gallons, eight five three. Then imagine you fill the eight gallon jar with water and your task is to figure out how only using these jars pouring water back and forth between them. How can you split that to exactly four gallons of water in the eight gallon jug and four gallons of water in the five gallon jug? Feel free to pause here and try to figure that out. So what does that have to do with this study? Well, Jeremy and Keane found that the groups who used chat GPT to help come up with ideas tended to just use it like Google. They didn't leverage its abilities to chat, to iterate, to serve as devil's advocate. They didn't engage in any kind of back and forth. The sort of thing chat GPT offers. That would have been very useful in coming up with ideas, solutions to problems. Instead, they tended to stick with its early often mediocre ideas and answers. And short, they only scratched the surface of its potential, and thus did not, though they could have, outperform the other groups. When, when human beings are given a task that seems like something they've done before, they end up settling called the instilling effect, which is basically they cease to search for better solutions. And even when a better solution is possible, and even when subjects are told that there's a better solution, what researchers at Oxford have demonstrated more recently with chess masters, is they fixate on the solution that fits their established paradigm, rather than even looking for the better solution. This is the instilling effect. Dunker studied it many do, but I love, I'd never thought about that, that even the chat interface itself, lends itself to this sense of, I've been here before, and the more someone feels I've been here before, the more endangered they are, paradoxically, a falling prey to the bias. Yeah, I've been here before. I've used Google to do things. I'll just use this as super Google. And that blows me away, because all the studies you mentioned, people will do this even at very high functioning levels. The chess masters, mathematicians and physicists will do this. They'll have a way that they have solved problems in the past that is laborious and difficult and requires an all day sort of thing at the board. And then there will be a much simpler easier, will not take all day solution to the thing. And they can know about that. I'm still going to do it the way I usually do it, because that's safer to me. What do you have to add on this key before we move forward? I think you said, everybody knows how to use Google. And I think actually part of the problem with a chat GPT interface is that it does look just like Google, like the search box, asking me a question. And that's the problem, because people use it like they would use Google or Wikipedia. And that's actually not the way it should be used. One of the CEOs of the companies that we partnered with had this great insight. And he said when it comes to chat bots, the emphasis belongs on the chat, not the bot. And so we have to develop conversational interfaces that allow us that encourage us to have conversations rather than asking it a question like you would Google. And I think that's still in the early stages of us designing for that. We all use bots before, and they tend to suck. And so I wonder if there's always kind of like there might even have people going into it. You were mentioning all these studies like people kind of expect what they have always expected. And I don't know how many times we've been on those kind of like chatbonds online, where you go like, you know, you kind of like have to almost guess the exact phrase that the engineer had kind of put in there. And so do you think there might even be kind of like a build in bias against getting a really, really good answer because we're so used for chatbugs, not business. I think it's a brilliant point. Kim, I was just saying that, you know, one thing I took from Kim, what he just said was, we've got to become fluent AI conversationalists. That's actually the kind of the capability gap that's got to be climbed here. And just by way of an anecdote, Henrik, I think it may address kind of your observation. I feel that what's required to become an AI conversationalist, so to speak, is actually to have a personal epiphany experience, like a deeply personal experience. And we found ourselves, Keon and I, we're, I mean, we were at one of the largest financial services companies on the West Coast. And the head of innovation we asked, so what have you been using chatGPT for? This person said they hadn't used it yet outside of our study, which is mind-blowing. And I found, that they were kind of imagining a future project where I'm, I'm planning on using it because I know that this innovation project. And to me, if you contrast that, by the way, I'll give you just a personal story because I think it's really illustrates the point. My grandma is approaching her 90s. I was with her over Thanksgiving. She was saying, hey, you're doing all this chatGPT stuff? Like, what is it? This technology. And I sat there thinking for a second, I said, okay, what's a personal, emotional question you want to ask a friend about? And she goes, I thought this was technology. I said, I just bear with me. Just humor me for a second, Cranie. We're in the car on like a four-hour drive. So I got nothing but time. And she said, well, I've been wondering about when it would be appropriate to move into a assisted living facility. And I, first of all, I'm like, whoa, she's never asked me, that's, okay. So honor the moment, this very kind of significant. And I said, okay, let's just, let's use this. I want to use the technology to show you about it rather than tell you about it. I opened up the chat interface and the, you know, whisperer, like voice. And I just said, hey, we're trying to make a decision about whether we should move into assisted living. Would you ask us three or four questions to get a sense for where we're at before you provide any recommendations? And then, you know, immediately it responds, you know, chatGPT, sure, Jeremy, I'd be happy to help you with this decision. First question, can you tell me about any changes to your mobility recently? And I looked at Granny, I'm driving. And I just talked to it. And she's, you know, well, no, for the most part, I mean, we're still getting around, you know, we love going to the gym in the morning and Papa loves to golf. And, you know, and I'm just, she's just kind of rambling, right? You know, as a human does, right? And I said, that's good. That's good. So I, I just, you know, hit the, you know, upload button. And then great, thanks so much. Next question, can you tell us about relationships with your caregivers in your life? And I said, Granny, answer, you know, we don't really have any caregivers. I mean, my daughter comes up from Dallas every couple months, but it's not, it's more just for visits. It's, you know, so this totally unstructured, like, it's the, this is the premier use case for experiencing chatGPT, right? Because a couple more questions come up. And then, boom, it's, you know, chatGPT says, okay, based on what, based on this conversation, here are a few thoughts for you to consider. And I handed her my phone. And I said, I don't need to read this. I mean, this is your conversation. Just take a look at it. And her eyes got wide and she was like, I didn't know computers could do this. I said, yeah, they can't until now. But what's crazy to me, and the reason I mentioned that story is because all week long, there are these moments where we're at the dinner table, we're in the kitchen or something, and she goes, she's kind of elbowing me. Hey, Jeremy, do you think chatGPT could, you know, help with the recipe? You know, she has all these ideas about, you know, call it application in her life. The only way, as a nearly 90 year old, she can have an imagination about ways to use chatGPT is because she had this deeply personal epiphany experience. And to me, and what, you know, Keon and I, as we're standing in this financial services company with this innovation leader said they've never used it, they ask us, what should, what should I do to get started? And what I found myself saying and what I believe is really the answer is, have an emotional, personal conversation that you would want to have with a friend. But importantly, ask chatGPT to ask you questions before you get started. And to me, there's something about changing the frame of the conversation, you know, Henry, to your point about, we've all interacted with dumb chatbots. Well, GPT is not a dumb chatbot, but you actually have to, in a way, you have to frame the conversation in a way that gives it the benefit of the doubt. And I think that's really the challenge for most users is they don't know they've got to give GPT the benefit of the doubt. So having some of those early prompts that empower a user to discover this kind of incredible interaction that's possible, I think is like a key point of friction that's keeping people from realizing potential. This is, I love this. I just did this because I'm working on a book about GDS. And I keep playing with GPT to see like, oh, is there anything in here that's fun to do? One of the things I did recently is I was said, okay, have this particular philosopher define this. And then what they, what would might they say? And then they gave a definition and I was like, okay, now have this philosopher challenge that. And, but take it into account and refine it. And then they produce theirs. And then I did that over and over and over again. And at the very end of all that, I said, now imagine Vittgenstein, who's the philosopher of defining anything. Look at the entire conversation we've had so far and just say, how would you sum this up in a way that challenges everyone who's already spoken, but also takes it into account. And I could iterate that a million times if I wanted to. And it just keeps getting more and more bizarre. It's like one of those AI images with like, make it more comfortable, make it more comfortable. And eventually there's somebody in a recliner and in the cosmos. And it was doing that with words. And, but in that space, I'm seeing all these different things that help me brainstorm. And so your study reminding me of this thing that already had played around with. And so I'm very excited that you put this to the, well, let's actually scientifically investigate. It was that a good use of, well, I was looking at it thinking, was that a good use of David's time? Even the way you're talking about using Chat GBT, we found is mostly foreign to folks. Most folks interface with Chat GBT like a Google search query, where it's a single query and a single result. And most of the time they go, wasn't that good? The technology's not there. And they kind of dismiss it. And that's kind of it. The vast majority of folks in teams have a cursory interaction where they enter a kind of largely uncontextualized prompt and receive a very mediocre response. And they go, I knew it wasn't that good, right? And even the description of you being almost recursive and going back and going back and going back, that's a behavior we saw that is exceptional. And it is the exception, but it delivers better results. What's wild is if you, if you mesh the call it performance with, and across that data with sentiment data, because what happens is, it's saying nothing right now about comparing with AI versus without AI, kind of setting that aside for a moment, which is interesting. But even the within the AI-assisted groups, the teams that approached AI like it's an Oracle loved it, felt great, and they underperformed. The teams that approached AI like a conversation partner, overperformed, but they didn't really like it nearly as much. In fact, it felt more like work than magic. And so there's this fascinating dichotomy, if you will. If you want it to feel like magic, chances are, you're going to get, you're going to derive a fraction of the value of the interaction. But if you want to derive maximum value, you actually have to invest effort and thought into the conversation, which all of a sudden doesn't really feel magical, it just feels like a different kind of work. And the truth is, I mean, going back to kind of instilling or satisfying or however you want to slice it, most folks aren't that interested. They don't care enough to push. And to me, what strikes me about your research study is, you care so deeply that you're pushing and you're examining and you're cross-examining. What we observe in many cases and organizations is, folks get a cursory, they provoke a cursory interaction, and they're satisfied with the cursory answer, in a sense, because that sense of care and also an understanding of the way effective interactions unfold works, it isn't really there. You know, people we found are satisfied with a good enough answer. And to really get better answers to Jeremy's point, you have to rethink your workflow and change how you as humans work together in person, virtually, and with Jeremy Bay. And reframing your behaviors takes work, like you got to go to the gym and work out. And that's hard. And a lot of, let's just say, middle-aged adults don't want to work out and reframe and rethink how they have done work for decades. And so that's really, really hard. And that is part of the work that's acquired, not just from learning how to use GPT, but in a human behavior, how do we work with it with each other? And so, there's amazing other research that came out this year that showed large language models can improve ideation and creativity, theoretically, you know, very, very high. So there's a big ceiling on how it can help us with ideation. What we found in our research actually was that, yes, there's a theoretical angle, but when you actually applied on teams in actual organizations with real problems, they're like here in terms of how they can actually use these technologies to develop better solutions to problems. And so there's like this big gap between what's possible and what's reality. And in order for us to close this gap, we got to change how we think, how we work, how we collaborate on a team, and how we use this technology. And all of that is absolutely brand new to humans within the last year. And I just asked the study show that when you had people just ask a question or post like a problem to chat GBT, and they didn't kind of like quiz it because they were lazy. They just kind of accepted the first answer. They didn't perform as well as other people. Yeah, exactly, exactly. It's not quite as simple as that. It's not like one thing that teams that don't have AI spend an hour working really hard exploring a huge kind of solution space. They have terrible ideas and they have great ideas. Teams that do have AI, instead of spending an hour kind of exploring like a broad variety of ideas, really quickly they get to like good enough ideas and you know, call it in 10 minutes. But what do they do with the next 50 minutes? They don't push farther. They go, you guys want to go get coffee, right? And so now when someone heard that, they asked me a question, well, if they only had a B idea, why didn't they keep pushing? And what I realized was participants in the middle of the study don't know the grade of the grading doesn't happen till later. And so good enough, you don't know if good enough is a B or an A or a C, right? It feels good and you go, well, this is, wow, none of us would have thought of that in 10 minutes. It's pretty great. Like I think we're mostly done. But that decision is made without respect to the broader field of possibility. And because the bar is kind of set sufficiently high by early output from JGBT, whereas with a human team, maybe there's himming and haing and struggling and there's fits and starts and good and bad. But the teams actually generate a broader, what we found is they generate a much broader variety in terms of quality of ideas. And they actually many times generate more ideas than teams with generative, you go, dude, all you have to do is say, give me a thousand ideas, it will do it. But you can literally just prompt it for volume and variation. And it would. The problem is teams don't approach problem solving that way. And probably also you have the built-in problem with an LM model that good is average, right? Like almost per definition. And so at least what I found it is that if you ask it, for example, give me a really shitty idea, it'll come up with something where it will say something like, hey, this will be highly illegal. So it will be a shitty idea. And you like, yeah, wait a minute. It's not that illegal. And you could also probably change it a little bit. And so you can actually make a pretty good idea by prompting it out and kind of like the perimeter of, of its ideation space. And I think one of the things that we found is that sometimes good enough is okay, depending on what the context is and what the problem you're trying to solve for. If the problem you're trying to solve for is not mission critical and you're optimizing for speed, then collaborating with AI to come up with solutions to this problem in 10 minutes might be good enough. And then you, and then you move on. But if the problem you're trying to solve for is mission critical, and you can't just have a good enough answer, but you have to have the right answer or the best answer. That's when AI has a limitation in that it oftentimes allows human teams to come up with just good enough answers. And that's really not good enough. The question on the on the premise of ideas versus the problems, obviously for somebody who deals in entrepreneurship all the time, one of our whole belief systems is that ideas comes the way after having a good problem to solve. And so as you were studying this and was thinking about how to phrase the questions, how much were you looking at problem identification versus ideation? Yeah, one of our, actually one of the partners in the study, we partnered both with US organizations and also European organizations. One of our European organization partners, he's one of the things he said to me in the past was oftentimes the problem is the problem. When we're doing innovation organizations, the problem is actually where we need to be focused. So Henry, you're exactly right. And one of the things we did in terms of the research methodology is at one point in the exercise, we actually forced folks to redefine the problem in as many ways as possible. So they were given a problem from a problem owner in the organization. They were given a prompt to generate some solutions and through a series of exercises. And then at one point, we actually had them almost backtrack and say what are all the different ways we can view this problem. So it's not exact analog to what you're saying from an entrepreneurial perspective, right? The individual is often the one digging in, validating the problem, getting familiar with users, pain points, etc. It wasn't, you know, we didn't have weeks and weeks for folks to undertake problem solving. It was more of a kind of a short sprint session. But even in that short sprint, we totally agree with your belief that the problem is really critical. And we carved out deliberate time in the study to make sure that folks explored other problems as a means of generating other solutions. Because one thing that I do, what it does seem that AI is good at, sometimes it can be difficult to kind of get your out of your own hat and look at the problem from a different vantage point. But obviously the nice thing about role playing with AI is that you can get it to role play any character that you want. And so I find this kind of reframing exercise that Thomas Adele wrote this cool book called Watch Your Problem and it really digs into, I think is that you can kind of, it's very difficult to do just by yourself by doing what David did and kind of like invent all these different personas and saying, you know, how do we look at the problem from this thing? You often redefine the problem so that the ideation becomes easier because you might not be solving a problem that's very difficult to solve. One of the things we actually found that a general AI can be very useful for is assessing those assumptions. Like we're having a conversation today about how it's difficult to forget what the problem is and that's because we have particular biases about framing that context or that problem that we're in. And so one of the great possibilities for a general AI is that it allows us to think outside our own assumptions and we can use it to think about like what are some counter arguments, what are some alternatives, what are some potential alternative scenarios that I even think about that would allow us to perhaps reframe what the problem is because again, we're biased going into whatever that context is to begin with. And now we take a break from our episode for a word from our sponsors. And now we return to our program. This AI movement is very similar to the mid 90s when a lot of us kind of like starting to use the internet and people will go, you're showing, you might, you know, I started back and fight on it, but then suddenly you're like the browser materialized and you were showing, I remember showing my mom the brush and you're like, what can I use it for? And you're like, you can use it for anything, right? And she goes like, well, can I check if there's a book available in the library? I'm like, sure, it's probably not like the thing that's going to transform the world, but it's still fascinating. I wondered, David, you studied this, you know, with people's psyche. What do you do when you need humans to talk to humans in a different way or learn to talk to humans in a different way? Is there tricks from human to human interaction that we could kind of use for us getting to know this new thing? The big, your big takeaway here, which I thought was really amazing. You have two big takeaways in the study. One was, don't look for answers, try to have conversations. This seems to lead to better outcomes in this particular ideation space, for sure. And we're so used to doing it the other way. Like I'm looking for one very specific outcome. I love that that's one of our big takeaways in the study. Try to have a conversation with this thing and you would be amazed at what happens. And this is also true when you're facing difficult conversations with other human beings. Try to become outcome independent and also you should be attempting to ask questions that allows the other person to articulate their position on the matter. And in that articulation space, that's where we will have like a better outcome in the conversation. If I'm specifically attempting to get to a very specific goal in a conversation with another person, if you get there, you've also achieved creating the world that you already thought existed. And you're not going to be surprised by a damn thing. And this also feels bad as the person who's on the other side of the conversation. You can do this with GPT. I've done it. It's incredible. You can say, hey, I'm wondering what you think about this. And then like, I don't know. Like you can really have one of these strange, they call it cognitive empathy. NYU coined that as a phrase where I'm sure there's a reason this person is acting, behaving, thinking, feeling in a certain way. And I have to have empathy for the fact that they are motivated, reasoners. And they have some sort of way out of seeing the world. And what I've more interested in is how they arrived at their answers than I am at the actual answers that I'm receiving. And if we can get both people in that page where, oh, wow, I've learned something about myself. This is why I'm arriving in these answers in this way. Everybody levels up in that one conversation. It feels good. I've been astonished at how GPT has biases. It will start to exhibit certain human biases in its conversations. But I've also been excited by interacting with it in a way where I get it to introspect. I get it to say, I wonder why you thought of it that way or I wonder what's leading you to that. And it starts to say, well, I bet that's it. And it'll start talking itself out of certain ways of seeing things sometimes the way a person will. And I'll say, well, actually, when you look at it this way and I'm like, wow, you're, you're a robot. Okay. So I got to tell you a couple of fun anecdotes that are totally, you just got me riffing here, David. One is, Henry, I think your question made me think of something Kevin Kelly told me, which Kevin Kelly is amazing artist, visionary technology, futurist. And I was talking to him about how he uses mid-journey and Dolly. And I said, what are you going for? And he said, I'm waiting to surprise myself, which I thought was a beautiful description of the goal. David, you said, like, most times we're trying to get what we already, what we've already premeditated. And if you realize your goal is not to get what you're looking for, but to get what you didn't know you were looking for. And to that end, I would say, one of the things that I've done recently, Henry has been involved in this project is we've built this coach basically to help people learn how to talk to AI. And we've got it inside of an organization where there's kind of a pilot group of users who are playing with it, but no one's gotten to the end of the call it, pre-programmed drills. And I had a question yesterday, I was like, what happens when somebody gets to the end? Because we only have so many drills. So I just really ordinarily, these drills, you know, take, you know, meaningful thoughtful attection. And you know, somebody, we imagine, so we do want a day or every couple days, right? Basically just drills on how to become a conversational. Well, I just kept hitting net like next, did it. Great. Did it. Great. Did it. Great. And I got to the end. And I was texting one of our partners going, what's going to happen when we get to the end of this thing? No kidding. The bot keeps creating more drills. I'm like, and I literally, I go, dude, my mind just got blown because it came up with a totally great trail that I never thought of. It's like, I'm not sure if any of you are familiar with the book, Why Greatness Can't Be Planned. It's an incredible book. But more so, I think a philosophy and it's obviously written about AI and about, you know, how we'll get to a artificial general intelligence and how we need to rethink it. But the premise of the book is that instead of pursuing a goal, we should pursue interestingness. And that becomes the stepping stone that kind of lead us into new discoveries. And then the skull will take care of itself. And it's kind of fascinating. David, you're just saying a little bit like, and I think what Jeremy was saying, the study was kind of showing is that if you're kind of like, not just trying to get to the idea, but you're trying to kind of uncover kind of like new pieces of interestingness, it could lead you to a much better place if you're just trying to get to like the the goal. Yeah, this is my experience. Like, you can totally use these bots to say, will you please define the word superfluous to me? And then you're just like, hey, here you go. And then, but you can just go from that one prompt to some weird places. Like, like, why do you suppose that is the word we use for that? Are there other ways of expressing it? And then it will also go, well, yes. So then you think there could be a better way of doing this? And all of a sudden, it's brainstorming like, oh, there might be a better way. And I'm like, who, and then you're like, has anybody else said that? There are so many ways to bounce even out of a pure definition into some bizarre ass places with this thing. And you'll walk away excited. And you're starting to fountain out on your own. And it's prompting you to the, and now you're doing this sort of like, oh, we're going to prompt each other for reverse prompting. Yeah. That's been my goal here recently. It's like, I want this thing to get me to a place where I feel like I just got prompted to ask better questions. It's prompting me. You were mentioning, you were mentioning something earlier about, you know, learning how to train. And so could you talk a little bit more on like, how did you, when we go to the gym, we probably all kind of know that we have to go to the gym. And we end up not doing it or this radio show once when somebody was asked, they had all these experts and going like, what is the absolute best kind of form of training? And then somebody answers the way the one you get done, which I thought was kind of like nice. And so when it comes to brainstorming or using chat to be tea as a partner in idea discovery, what is the training methodology that people should apply? So I love this gym example because we know how to work out because there's coaches there who have done it for decades. Right now, we're in a situation where the coaches don't know how to train people and the coaches have to be re-coached and retrained. And that's what's coming out of our research findings. And so to answer the question of like, how should teams think about using generative AI to come up with more effective solutions to a problem? We found that there is a commonality in terms of like how we design innovation and ideation. And we come up with this very basic five step, and the one that I think will help people remember how they should think about using generative AI on their team collaboration activities. We call it FixIt, F-I-X-I-T. And very quickly, F stands for thinking about having a very focused problem. So you don't want to boil entire ocean, you don't want to ask chat GPT, hey, how do I improve my sales by 10%. Well, that's just way too broad. You have to be very, very narrow to ask a question that can allow it to open up dialogue around a narrow point. So you won't have a very focused area to think about. The second part, which I think is really critical, is that you have is the eye. You have to have individual thoughts on your own as a human before you go to chat GPT. So you've got a small focus problem. It's important for you as a team member to think on your own. What are some solutions I might offer? Because we're all going to have different solutions. The next step is then to go to chat GPT or a large language model. And it's X, provide context. So you need to give enough information in order for a chat bot to ask you the right kinds of questions to have the conversation to help you get to the right point. If you don't give it enough context, it's just going to give you generic information that's just maybe not even good enough as we'll be experienced. If I made there, Keon, one thing that's that's very unexpected, but in getting to the kind of the conversational dynamic, if you're not sure how much context to give, ask chat GPT. Ask how much, let me know if you need more context to be able to prompt you well, right? This is even that is I think it flies in the face of much of our kind of dogma of, I've got to be the one to decide, you know, or like custom, you know, Henrik and I interviewed Dan Shipper about custom instructions, right? And we said, well, how do we, what if you don't know what to put in your custom instructions? You know what Dan said? Ask chat GPT to interview you about what it should put in its own custom instructions, right? But so that, to me, that getting back to this idea of context, it's one thing if you say, I've got to prepare the perfect dossier that I'm going to upload. It's another thing to say, here's my goal. I can give you any context that you need. Can you tell me the pieces of context would be useful in helping frame this and narrow this appropriately? So really, it's way more conversational than even, than we knew even at the time, I think, of the papers writing probably. And so once you've given a context, the next part is the part that we've been learning about is you have to have an iterative conversation with it. As opposed to going to like an encyclopedia, you want to treat it like a friend or like a colleague and have a back and forth conversation with it. And this was really, really critical. At that point, once you've done this iterative conversation, then you go to the fifth step, which is the team. Then you bring this to the team and say, okay, I've had an individual interaction with chat GPT around this particular problem here, particular solutions or ideas that we were developed. Now, let's as a team, you've all done this individually as well. Let's think about, let's put this on a whiteboard or virtual whiteboard. Let's prioritize what we've seen for the resources that we have and the time that we need to solve this problem. So it starts with very narrow problem. Then thinking about it on your own, giving it context so that it can give you the right kinds of conversations that you can iteratively have with it and then going to the team and having a prioritization activity that you're exercise for the team decide that's the right thing or we should really focus on that. Or the team will say, this is interesting. Let's do another loop because we found an interesting pathway. Now let's iterate that even further. So let's go back. And so we found this this this five step process called FixIt to be a elegant solution for humans to start thinking about how they can bring AI in as a co-pilot on their team to come up with more effective solutions to problems that they face. FixIt, you can go to howtofixit.ai to see the whole deal. And here it is, in summary, F set a focused problem, be precise rather than abstract. I individual ideation first safeguard individual human creativity. X protect your context, train the AI. I have interactive conversations. AI as a collaborative thought partner and team, team incubation facilitate your decision making. Each one of those has several steps, lots of explanations and you can check it all out at howtofixit.ai. I love the idea of like coming to this and not trying to answer the question, trying to get it to show you the thing that the answer I didn't know I was leaving looking for or just to be surprised by it. That's a great use of a new technology. It's a new technology. What if we did new things all the way around with this guy? I love it. I had that epiphany that you were discussing earlier, Jeremy, with the I my epiphany was asking GPT about epiphany's. That's when I realized, oh, this is a powerful tool. I just pulled it up. I have it over here. I asked it. I just talked for like 15 minutes into a speech to text about tear msue moments, which is an idea I had for a sub stack, like a post or maybe something. I was like, I would like to maybe write about this. Here's what I'm thinking. It's because the first time I had tear msue, I didn't know tear msue existed and I was at a conference and they brought it out as a dessert and I was like, damn, this is good. What is this? Everybody was like, I was tear msue and I was like, oh yeah, yeah, tear msue. I love this stuff. But I learned it existed. I just had a accommodation moment as they'd say in psychology. I now entered a universe in which tear msue exists. Before that, I wasn't in that universe. Now I am. This is something I was like, I bet there's a way to write about this that would be fun. I told this story to GBT with a bunch of other things and then I said, could you now give me from the perspective of a bunch of different famous thinkers? What would they say about this? Then I asked them to have them argue with each other and disagree. Then it worked. From that point forward, I started every time I gave it one of these ideas, I asked it, hey, tell me, have all these different famous people disagree with me. Show me where they would be like, maybe you haven't seen it this way. That started becoming like a very powerful way to to get started on something. I found that that was like an accelerant to whether or not I want to pursue something. It made me more excited about that. You think they would squash it? Absolutely not. Every time they disagreed with me, I felt like, ah, okay, now I really know what I want to say about this and that's been my epiphany moment. I love this. I love that everyone in this conversation has been excited about this. So many conversations about the technology have been like, well, so anybody see Oppenheimer? I do not deny that this is a tool and a tool could be used for all sorts of things, but I'm excited that people are using it for good things and we're doing research right now and your research is in the domain of behavior. How will human beings interact with this? What are the pros and cons and how could we get better at it? I commend thee. Thank you. Thank you. You know, we're pretty stoked because we heard from Harvard Business Review. They're going to actually feature the paper of some of the most interesting research that's been conducted in the last publication cycle. So hopefully by the time this episode is out, they'll actually have, it will be hitting new stands as well. But if for whatever reason, this is either before that or it doesn't happen, if folks can go to howtofixit.ai if they want to pull down the paper and they can grab some of our early recommendations for teams who want to implement this model. Howtofixit.ai. But what the one thing that just occurred to me as well is I think similar to the chatbot phenomenon of, oh, we've been here before. I think there may be a, there may be a lack of appreciation of the nuance and ability that that has to be present in the organization, right? So whether it's investing in conversational ability, whether it's having outside facilitation, that's one thing we've seen actually, right? Is that teams that have outside facilitators help them interface with AI tend to perform better, right? So just like you don't want to approach the chatcpt window like a Google window, don't approach the task of ideation to come full circle back to the beginning of the conversation. Don't approach the task of ideation now in this new world of generative AI like we've been here before. Think about enablement, think about abilities, think about facilitation, and your team is going to get exponentially more benefit than if you just do it like you've always done it. That's a great point. Don't get me going. I had already wrapped things up in my mind because you like if you're having an argument, if you're trying to discuss transportation and Olive River had our wagons and horses, your discussion of transportation is the very idea of how to have a discussion about transportation is limited by that space. And then when you have all this new technology, a discussion about transportation doesn't even look like a previous discussion, what a bit about transportation. So ideation is a concept that will itself evolve thanks to this new tool that is a really cool takeaway. I dig that a lot. And I ask a final question just because we're here and when do you know when you have the idea, when do you have this kind of contentment moment that David just expressed with the conversation, but as you're ideating? Is it isn't there a book on that? Henrik, I think, called the acorn method? Oh, you're just flirting. That is my book, which I couldn't recommend reading, but like no, I don't think that'll end. No, no, no, no, tell me about this acorn thing. The acorn method is a book that I wrote about how when you are trying to grow your business by growing new businesses. And the thesis is that companies should see themselves as a forest not just becoming a very big tree. And so as you are trying to see how do you not, how to make sure that what happens to most of the Fortune 500s, which is that the halfway house is getting shorter and shorter. How do you kind of like break that mold? The way that you do it is that you look at the way the trees have evolved over many hundred thousand years. And you say, well, instead of just trying to grow, grow, grow, we should look at how do we drop small acorns around ourselves and how do we make sure that we make sure that they inspire and become maybe a bigger tree than we are. And if you look at the Googles and you look at the apples, that's exactly the methodology they have completely stolen from the world woodweb trees. If you want more wood, drop more acorns. That's, that was my takeaway from Hendrick's book. And I didn't mean it tongue in cheek when I answer your own question with your own book, Hendrick. I think the way you, you have that at 50 moment, it's not, it's not that the job is not done with ideation. The job is done when you implement a desirable, infeasible, and profitable solution. And the way you discover which of those works is you drop acorns. You don't put all your eggs in one basket. You try lots of things in a low resolution, high speed, high velocity, kind of manner, that's entrepreneurial in nature, that's resourceful, that's resource constrained, et cetera. And over time, you start to see which of those trees grow. So there can be this kind of artificial kind of declaration of victory at the end of an ideation session. The truth is, the next critical capability is around new business development. And the way you learn which of the ideas works is by trying a lot of them as cheaply and as quickly as possible, right? And so it's not that you get the answer really quickly so much as that you get into the process of piloting and prototyping more with more confidence. I would also say that you know an idea has worked when at the end of the activity or the exercise, you've got a smile on your face. And it makes you happy. As I have been this entire hour plus talking to you guys, smiling, grinning because it's been just very enjoyable, but also it's been a great idea for us to get together and have this conversation. So thank you so much. That is it for this episode of the You Are Not So Smart Podcast. For links to everything that we talked about, head to You Are Not So Smart dot com or check out the show notes inside your podcast player. You can find my book, How Minds Change, wherever they put books on shelves and ship them in trucks. Details are at davidmacranie.com and I'll have all that in the show notes as well right there in your podcast player. My homepage, you can find a roundtable video with a group of persuasion experts featured in the book and you can read a sample chapter, download a discussion guide, sign up for the newsletter, read reviews and more all of that at davidmacranie.com. For all the past episodes of this podcast, go to Stitcher SoundCloud Apple Podcast, Amazon Music, Audible Google Podcast, Spotify or You Are Not So Smart dot com. Follow me on Twitter at davidmacranie, follow the show @notsmartblog. Also on Facebook/You Are Not So Smart. If you'd like to support this operation, go to patreon.com/You Are Not So Smart. Pitching in at any amount gets you the show at free and extra episodes from time to time and at the higher amounts you get boasters, t-shirts, sign books and other things. The opening music that's clash by caravan palace and if you really want to support the show, tell everyone you know about it and if there's an episode that really lands for you that really is useful that really moves you, that's the one you share. Please share that. Check back in about two weeks for a fresh new episode and you can check out all the things we discussed on this episode at howtofixit.ai and you can catch the podcast that this will be a episode upon at BeyondThePrompt.ai.