Experimenting with AI responsibly and transparently with Economist Impact’s Jeremy Kingsley

It should be clear to customers what you're doing with the data. Then on top of that, there are going to be much more stringent requirements from a regulatory perspective on when data is actually used to make decisions. And that in a finance context becomes particularly challenging and companies have found themselves in a whole water a number of times. Welcome back to Conversation with Send Esk, where we explore new technology and trends in customer experience. Each episode we speak with industry innovators and experts to hear their thoughts, unpack industry trends, and discuss the most important ideas around CX. I'm your host, Nicole Saunders. Today's conversation is with Jeremy Kingsley, head of strategic foresight at Economist Impact, the in-house think tank of the Economist Group, publisher of the Economist newspaper. He also leads Impact's research on technology and innovation. Jeremy advises and helps organizations navigate technological change and its impacts on society. And has spent more than 15 years covering technology trends, innovation policy, and business issues as a journalist, researcher, and consultant. His writing and analysis has appeared in Wired, the Economist, and the Financial Times. Coming up in this episode, we explore several topics related to AI, some of the ways that jobs may evolve, how to experiment with it responsibly, and the need for transparency around how companies are using artificial intelligence. Stay tuned to hear our conversation. Ready to take your customer experiences to the next level? Build lasting relationships, the Zenus complete customer service solution so that you can exceed every customer's expectations. Sign up for a free trial at zendesk.com. Jeremy Kingsley, welcome to Conversations with Zendesk. How are you today? Good, very good. Thank you. Thanks for having us on. So glad to have you here. So you do a lot of research and writing in the space of AI. I'd love to have you tell us a little bit just about what are some of your areas of focus that you have been really thinking and researching about over the last couple of months. I have two hats, one of which is leading our work on Strategic Vorsight, which is all about helping organizations think about the future. And how it might play out. And the others that I lead our research on tech and innovation. And so there is no greater tech and innovation trend that is impacting on the future at the moment than AI. So a lot of our thinking at the moment is focused on that. How the technology is developing, how it might change our economy and society and what we should do to govern and shake its adoption. I understand from some of our previous conversations that financial services is one of the areas we've seen. Some of the earliest innovation and adoption of AI technologies. Can you tell us a little bit more about what you're seeing there and why is that the industry where that is starting to kick off the most finance has always been a industry that's been an early adopter of AI and technology in general, right. Companies in that industry already have lots of data. They employ lots of data scientists and they're working on these sorts of problems, the sort of problems that AI can help with, like fraud. For a long time and there's incentives to do so. There's, you know, to save money to stop losing money, to crack down on money laundering and things like that. These are things that, you know, 90% of money laundering goes undetected. And so AI is hugely beneficial at doing that when you can spark happens and you can start to see how things are different from that. So you see a lot of use and you have done for several years now in fraud. Sorting legitimate from fraudulent transactions, you can draw on decades of data on customer purchase habits, etc. And all of that is saving banks money and also offering better customer experience, right. These are not blocking as many legitimate transactions as you go along. And there are other ways as well, right. They're using data from customers to predict credit risks, you know, if you're using non-traditional forms of data to assess credit worthiness of people and give cheap loans and then, you know, dipping their toe into a generative AI like everyone else. Are there things that you have seen come out of that industry so far that have had ripple effects for other sectors or other kinds of businesses? AI has been used in businesses across the world for a long time, right. It's been social media feed since you know product recommendations and translation, voice recognition, etc. And it is now capturing the attention of a lot of different businesses who are not as data heavy because of tools like generative AI that open up all this unstructured data for them to play with. And this presents itself as a bit of a more obvious opportunity at risk for businesses to take advantage of. So we're seeing a lot of businesses thinking about how they can take some of these generative AI models complement it with their own data right by insuing those models making models that are better suited to their particular purposes and each purposes maybe tailored to you understanding the jargon or processes of what they do. So we're seeing in visually a lot of companies doing that, but also a lot of companies looking to cater to the particular needs of different industries. So we see that mostly in sort of back office functions within businesses that's what we see most of the activity most of the job disruption in the short medium term. So thinking about the financial services sector, we're dealing with money, right. And so I understand that there are really high needs for data regulation and privacy in that space. Tell me a little bit about what is happening right now around data protection and regulation in the AI space. Well, I think we're seeing a big shift in the way we think about the data we share online, right. I mean, just in a consumer context, people have been sharing data in return for free services, social media accounts and email and free maps and things like that. But now there's sort of different kind of exchange going on, right. Companies to sort of open AI's, the large language models of this world, they're taking your data to train these AI models that you don't necessarily get an immediate return from right to those of those services that you have to pay for. And people might worry that they may be even sort of automating them out of the job, right. But it's ability to imitate what you do. So value exchange is like a really interesting question that means the regulators are waiting out to think about, well, you need to be clear on and transparent on how you're using people's data, right. I mean, consumers are waking up to but regulates the playing a role in ensuring customers have a meaningful choice on how their data is used and that companies are transparent in what they're doing. Have you seen any companies that are doing a particularly good job of being transparent with those things because I imagine every organization is trying to figure out what is the best practice. Any examples that you would point to you in the industry. I don't think it's rocket science, right. To be clear on what data you're using. To just to say whether this is where data that we're going to be using to train an AI model or whether you're using an AI bot and you need to disclose that. Then on top of that, there are going to be much more stringent requirements from the regulatory perspective on when data is actually used to make decisions. And that in a finance context becomes a particularly challenging and companies have found themselves in hot water a number of times. Apples, an example of this apple was accused of gender bias in extending some of its credit lines that are found to be offering more credit to men than to women. Right. And that actually comes up against a pretty clear set of regulation. Companies are not allowed to discriminate existing laws cover that. Right. But the trouble is why an AI makes the decision. It does. And it might not even be clear to you the providers of bad AI. The bias is in the data. But from a regulatory perspective and a consumer perspective, you want to have a recourse to understand how a decision is being made and to make sure that it is fair. And that's a big part of what regulators are thinking about when it comes to AI. It's about ensuring that the process is transparent. And it's also explainable if things are going to be these high risk applications or high important applications like credit scoring. That's a great example. And I think it points to the need to have checks and balances with these technologies and make sure that you have QA processes. Even if you're not as in as highly scrutinized a field as financial services, we do want to make sure that we're checking the outputs of these technologies and that they're correct in their serving our customers appropriately. Your example of Apple makes me think about how this impacts companies that are global, right, larger organizations. And I know that there are different kinds of data regulations in the US versus the European Union, even versus some of the country in Asia. Can you speak to me a little bit about what some of those approaches that are from different countries and different regions of the world. The pace of development of AI and the diversity of risks that it presents, poses a huge problem for regulators, right. So there's all spectrum of risks that they're worried about from bias and data privacy and security through to job displacement, misinformation, history to big questions about existential risk. And so regulators are scrambling with quite a few different responses to some of these questions, including data protection. One of you is, you know, we have a lot of laws already that govern this. We don't need AI specific legislation. We have laws against discrimination, we have laws to it. in trying privacy doesn't make sense to regulate a technology specifically. And so a lot of that does cover some of our existing thinking around the protection that companies will be used to do the same things broadly apply. On the other end, you have more AI-specific legislation. That's the approach of the EU. And so there's an AI Act, which has been years in the making and is nearing completion now. And that is regulating by levels of risk. So if you're entering as an organization, you've got to be thinking about what level of risk is relevant to your operations. Within that framework, the highest risk applications of AI are just going to be banned out, right? That's things like pretty quickly saying your emotion recognition and things like that. But there is a category of high risk applications where there is a risk to safety or a risk to human rights, as they see it, that AI's use and data's use has to meet certain requirements of around transparency and explainability. So those things like hiring and credit scoring and decisions all impact people's health. All of those decisions can't be made exclusively with an AI without a high standards of explainability and transparency. And that is given the way the technology works. Actually, a pretty high bar to reach, just because it's inherently quite difficult to look under the hood into the black box and understand how the technology reaches the decision. So that's going to be a big impact to when it comes online in a couple of months, maybe that companies will have to think about what they're doing. When it comes to thinking about it from a business perspective, you've got to wonder what legislation you're going to be having to adhere to if you're a global business. There are different regimes around the world. Previously, it's been the case that the EU has a, what's called a Brussels effect that has influenced the way companies operate right around the world. Companies just find it easier to just adhere to the strict dysregulations so they don't have to have different products in different markets. And regulators tend to actually just follow the model that the EU takes. But there's a lot of controversy around it. There's a lot of feeling that maybe the EU is overreaching with some of this legislation. Maybe they'll go too far. And you can end up with quite a fractures regulatory environment, which make it quite a lot harder for companies to operate. Because you might want to train your models this way in one place, have these transport to requirements somewhere else. So it's something that companies are going to have to be paying a lot of attention to. And it can be a bit of a regulatory headache. Sounds like it. From what you have seen so far, do you see that businesses are taking these things into consideration or is the excitement and the momentum behind AI driving people to start to implement things and then have to go back and review everything to make sure it's been done properly and securely? Well, you've definitely seen this land grab for data, right? And that is getting some companies into hot water. You can see the value that it can present. But you've seen what happened to Zoom. They got into a lot of trouble recently for revealing that they were going to use the data, the transcripts from your private conversations to help train its AI model for purposes that were not even clear. I don't think even if they would be clear how they might plan to use it in the future. And that caused a pretty strong consumer backlash, right? And they walked it back quite quickly to their credit. So you can see that there is a line that consumers are drawing themselves, let alone where regulators want to step in. Generally, the consumer feelings and the things that you should be paying attention just as much to. But overall, the regulatory principles are going to be clear on this as well. It's pretty clear you just need to be transparent on how AI is used when content is going to be AI generated. When people are interacting with the robot, particularly if a decision is going to be made by an AI. And if you're going to be using your own data, you've just got to be aware of things like the bias that might be inherently there, and whether you might be discriminating as a result. So of course, a lot of this sound like the kinds of things that companies would service in a terms of service. But we also know that most people don't read those super closely. They clicked, yeah, OK, except let's go and use the technology. Do you think it's important that companies find other ways to make it more apparent, for example, marking on a chat, but hey, you're talking to an AI right now that this isn't a human or those kinds of things? What do you think is the level of transparency and consumer friendliness of that transparency? What seems to you to be the right level for that? I think just intuitively you want it to be not hidden away in the terms of service. And the way the regulator said about it is that it needs to be a meaningful choice and it needs to be meaningfully made apparent. It can't be hidden in the terms of service that you're doing these kinds of things. And certainly if you're going to be taking data from others. So yes, you want to be crystal clear on when these things are being used in that way. It sounds like organizations may have to start to develop new teams and new competencies around how to regulate how to tune QA, these technologies. Have you seen any of that kind of activity happening so far or are most organizations leaning into their existing legal teams and things like that right now? There was certainly seen that the compliance aspects are a big headache for companies, particularly multinationals that are working across borders. And these things not later on to chief data offices and chief legal offices working together on these issues. So I wouldn't say it's providing new teams, but it is top of the agenda. Through your reporting and through all of your research, you get to interact with a whole bunch of different organizations that are using AI. What are some of the most compelling or interesting things that you have seen to date? I think we're still at this really early stage with AI adoption, right? I think we're seeing companies take it off the shelf that's figuring out their experimenting and actually very little of that is still public. At least when we're talking about the gender of debate, AI that is capturing everyone's attention. I think it's interesting and I'm going to be more disruptive when we're thinking about generating new products, new business model and things like that. And the examples where you see that it seems to be where companies themselves are more under threat by the threat of AI right now, I mean to pivot what they're doing to create something new. So there are several examples in the education space. You've seen companies like Jua Lingo, who created a learning platform for learning languages. They felt that they're under threat from these gender to AI models and they've pivoted to create a virtual tutor. Khan Academy has done a similar thing. And in fact, I don't know if you know the learning platform Cheg has a terrible name, but it seems to be basically a platform for cheating on your homework. But that used to be a platform that was allowing students to pay to have someone right there to their paper. And that was something that was completely thrown into crisis by the advent of chat GPT because why would you pay for that when you couldn't just use chat GPT to write a exam paper for you. And so they lost 50% of their stock value and disappeared into the ether until they realized that they've got all this incredible data that they can work with, right? They've got years and years of creating term papers and answering questions from students. And so what can they do with that data to create a kind of virtual tutor? And you think about how the education system might be changing and that actually, and papers are not gonna be house students can be assessed what really matters is new ways that they will learn. And so they've set themselves up to actually ride that wave of disruption in a more interesting way with their eyes. All those companies who are thinking about how they actually rethink quite more fundamental business models and generate new products is gonna be where we see more interesting innovation. - So it goes a lot beyond just gaining new efficiencies, making chat butts better and really, as you said, fundamentally rethinking how you're operating and how these technologies might influence that. - It's gonna be really interesting to see in time how we re-engineer industries around AI, right? I mean, if we think about big systems like education or healthcare, these systems that have been quite slow to actually transform as a result of technology, we have this incredible technology that can, you know, in education to use education again, deliver personalized learning, right? Understand individuals needs, their learning gaps, they can understand their learning styles, but our education system isn't quite equipped to deal with that, right? It still teaches in front of students and in classrooms that look the same as they did 100 years ago. So some of this disruption that we're seeing, you know, realizing that we need to change how we do exams might jolt us into rethinking about the bigger picture a bit and that's gonna require innovation around not just the technology, but the way we think around the coordination of the wider system. So that's what I'm really excited to see what we will actually do in areas like education and healthcare and things like that in the future. - It sounds like there could be a lot of really significant ripple effects from the technology in terms of not just how an organization operates, but really how the whole system works. You know, we've heard a lot of different concerns about AI, some of them are real, some of them are exaggerated. Based on what you're seeing, what do you worry the most about in this space right now? - I think that very real worries that pre-exist to the conversation around generative AI are around bias and discrimination and privacy, security. that shouldn't be treated any differently now. And I think we know what. to do about those and we should act on them. I'm less concerned about the bigger risks that people talk about existential risks and robot takeovers. I think we're a long way from that. I'm probably most concerned about the stuff that's in the middle, which is around misinformation and disinformation. I look at the power of the technologies around generative AI. It is mostly in the ability to make stuff up, to just generate content at scale and used in the wrong hands that can have extremely worrying effects. I think that's probably where regulators need to speed up the most to pay attention with the election. In the US coming up and elections around the world, this could be a really powerful force to reshape our world in some pretty nasty base. It would be a road or ability to tell good information from bad. Do you worry about that in your space as a journalist? I'm not so worried about jobs and disruption. If you look at the way technology, automation usually goes, you tend not to see these large employment shocks. Jobs evolve, new ones emerge and the net effect is pretty minimal. AI will come and automate a percentage of your job and augment your job in interesting ways and probably actually make things more interesting. I think to your question about the publishing and information industry, there's a lot of interesting stuff going on there. People will always value facts and original reporting, which is something that AI will still for now struggle with on the ground reporting. I agree with you. I think that those technologies really do have the potential to make a lot of jobs more interesting, better. My hope is that many organizations will use the efficiencies that AI can drive to free up people to do more meaningful work, maybe do more complex work, expand the capabilities of the business. It's not just about cost savings, but really about how can you take the workforce you have and do more with it. Something that you see all the time, right? If an AI boosts your productivity, you can cut workers or you can use this to produce more. If you make something more efficient, we tend to actually do more on those jobs change and they get usually at a higher order of creativity. And hopefully you'll see that more jobs will emerge to the future. As I said, that is the way it usually goes. If you see, even when we can automate things quite substantially, it doesn't necessarily mean that we do when ATMs were first introduced to the 1980s. It put bank retailers out of a job, but actually the number of bank sellers increased because banks found that they could open more branches because it was cheaper to do so with an ATM and they would have tell us to do the more complex higher order tasks on site and so you actually ended up with more of them, right? The same is true of accountants, financial analysts. We have many more of them than we used to. Decades ago, despite the fact that our work has been automated. So work changes, I don't expect AI to be any different. There was excellent examples and really great points. I hope that leaves everybody feeling hopeful. I know that it's been a big question in the space of customer support. And a lot of people are wondering, is AI going to put support agents out of work? And I certainly am increasingly seeing a lot of talk about, well, how can we repurpose the agents that are doing routine tasks and bring them into the more complex things? Because there's always going to be a level of engagement that the bots can't do and that you're going to need a human touch for. Certainly in that context, right? I think that's absolutely true. When I'm saying that's always the way it's been, there is an assumption that's the way it's always going to be. There is a question of whether AI is different, right? There is more and more that a computer can do. There's less higher order, more complex work for us. So maybe the negative use that we hit a ceiling, right? But even though our human needs are infinite, our capabilities aren't in some way. I think one of the reasons it's unnerving is that you can see which jobs are at risk, but you can't see clearly what those future jobs are like. So I realize it creates uneast people, but the historical precedent should be that jobs will change and hopefully get richer and more interesting. But we'd have to shape them to do that. I mean, another thing to say about job disruption is that even if we can't automate jobs, doesn't mean that we will or we should, that values come into play as well. Right? We can automate much the work that nurses do and teachers do very easily, but we choose society not to do that because we value the human touch. There's a sort of coaching element or other aspects to that that are important to us to our society. And so it's up to us to shape it that way. And in fact, you also see it in the right strike in the US. At the moment that there's a role for unions to play in organizations like that to fight for the importance of the human aspect to what they do. So that this AI is a big component in the right strike at the moment. So people will fight for their share of certain spoils of our nation and we as a society will shape the jobs that make AI work for us. It's key to remember that we do have a role to play as consumers and as workers in shaping how these technologies come into play in our space. So we've talked a bit about different industries, the direction of AI is going, some regulatory pieces. A lot about how this might disrupt jobs. This is a lot for businesses to be thinking about. If you were in a room full of industry leaders, they were asking you, what should we focus on? What should we be thinking most about? Are there any key things that you would recommend people be making sure they're paying attention to as they're thinking about bringing these technologies into their businesses? The one thing they should be doing is experimenting with this technology. I think there is some disruption that is going to be inevitable in all sectors, but they should be doing it carefully and responsibly as they do it. Generative AI particularly throws up all kinds of opportunities to throw unstructured data at it and see what you can do with it, see what it can reveal. You should do that. Think about how your business could be disrupted from first principles with this technology. But ultimately, the innovations that you come up with are going to have to be useful or don't bother with it. I think there are some limitations to generative AI in particular that we will hit. This technology still has huge limitations in hallucination and so on. That's a real problem. But from the regulatory angle and in terms of thinking about doing this responsibly, I think as we discussed, there are some pretty cheap and pretty clear things that they should bear in mind, which is to be transparent and to be clear on how AI is used, when it is used and be aware of the quality of the data you're using. Be respectful of privacy, ensure that data capture is opt-in, people are showing an increasing resistance to being scraped and so on. So I wouldn't build a business that relies on scraping your data on the slide without permission. So it sounds like what we need to be doing is experimenting, doing it responsibly, doing it transparently and being ready for some bigger seismic shifts that are going to be the downstream impacts of these technologies. 100%. Well, thank you so much for joining us, Jeremy. This is going to a fascinating conversation and I look forward to speaking again sometime in the future. Pleasure. Thank you very much for having me on. It's a great conversation. In our next episode, I'll be speaking with Desendask's own palette, "Chafel" about how IT leaders are approaching AI and what we've learned from a recent survey of IT leaders around the topic. If you enjoyed today's episode, please share it with a friend or colleague or you could write us a review on Apple podcasts. Thanks so much for listening and for being a part of our community. You can always join the conversation at zendesk.com/community or connect with the Zendesk users through our user group meetups. Find one for you at usergroups.zendesk.com. Until next time, I'm Nicole Saunders for Zendesk, the intelligent heart of customer experience. Build lasting relationships, the Zendesk's complete customer service solution so that you can exceed every customer's expectations. Sign up for a free trial at zendesk.com.