Speaker 1Hello and welcome to Don't Stop Us Now AI Edition.
Speaker 2I'm Claire Hatton. And I'm Greta Thomas. AI has been described as the most transformative technology since the harnessing of electricity. And we are here to keep you in the loop for what you need to know and do in order to stay relevant in this fast changing world.
Speaker 1Each episode, we bring you leading experts and together we explore how you can stay ahead of the curve and in the know for what skills will be valued in the future, how jobs may change and how industries will evolve. So why not subscribe to stay in the
Speaker 2loop and without further ado, enjoy this week's episode.
Speaker 1Hello and welcome to this week's episode. If you've been hearing about companies having AI agents working autonomously in the background, then there's a question our guest today is pursuing the answer to. Who or what is actually checking that those agents are doing what they've been asked to do? And that question sits right at the heart of today's conversation. It does indeed. And it's a pretty big question.
Speaker 2It certainly is. So our guest is Professor Sveta Venkatesh. She's a Deakin University Distinguished Professor and co-director of the Deakin Apply AI Initiative. Sveta's been named one of the world's top 15 women in AI and she holds one of the highest research honours Australia can give. Across her career, Sveta and her team have turned their pattern recognition and computer science model building work into genuinely hard problems. And they've looked at trying to solve problems to do with autism to high stakes security scenarios. And those things have evolved into real tools, patents, as well as a listed company.
Speaker 1Yeah, absolutely. And what makes this conversation particularly significant to us is Sveta's focus on something called assurance. Generative AI has changed her field profoundly and she's refreshingly candid about how the safeguards meant to keep AI trustworthy are running well behind the technology we're already using every day.
Speaker 2And that is alarming to say the least, isn't it? Certainly is. But in this conversation, you'll also hear, what AI assurance actually is and why it really matters. How little real oversight exists for AI agents that are already up and running and working out there right now. The unsolved problem of AI agents quietly colluding. Why the film 2001 A Space Odyssey kept cropping up during our conversation. And how Sveta's team is teaching AI to talk to children safely. While the frontier
Speaker 1companies and many corporates are racing to implement AI as it's developed, Sveta is doing the work that's not getting the same limelight or investment, but is arguably more important. Figuring out how we know that an AI product is what it says it is, as well as enabling us to detect when AI deviates from its brief and guardrails. Enjoy this thought-provoking conversation with Professor Sveta Venkatesh.
Speaker 2Professor Sveta Venkatesh, welcome to Don't Stop Us Now AI Edition. Thank you very much for inviting me to be on the show. It's really great. And frankly, a bit of an honor to have you. It's not every day we get one of the world's top 15 women in artificial intelligence on the show. So thank you for making the time. You might've heard that a question we ask all of our guests to kick off our conversations is what we call the dinner party question. And it helps us to our listeners sort of just get a feel for a bit about what you do. So if you were at a dinner party, Sveta, and you're sitting next to someone you hadn't met before, and they turned to you and they said, so what do you do for a living? How do you typically answer that question?
Speaker 3It's an interesting question because I get asked that quite often and you won't have to answer it in the simplest possible way. And when I first came to Geelong, I was at a barbecue and somebody asked me that question, what do you do? And I said, I find people who are very, very good at patterns. And he said, in what? And I said, in data. And he said, can you come to my office tomorrow? Because he turned out to be the CEO of Bowen Health, this big partnership on finding patterns in medical data. So effectively, that's essentially what I do. I find patterns, weak patterns, patterns that have some meaning, hopefully to some end. And that in a nutshell is basically what I do.
Speaker 2Yeah. And it seems like pattern recognition has been your thing that you're truly passionate about. We don't hear it talked about much as a skill to develop. Would you say it's a math skill or a data science skill or an analytics skill or a bit of all of them?
Speaker 3So it's always been a discipline in computer science, and it's just been called different things over the ages. I think the closest one is it's in computer science and statistics. And from that, what the pattern recognition then developed into a field, many, many fields, multimedia, which is, for example, how you search on YouTube today, the algorithms that you use for searching and indexing and retrieving were all developed in that particular field. It then became after machine learning, how you can make machines learn from data how to extract patterns. And now they call it machine learning. So it's a lot of different things. And I think it's a lot of different things. And I think it's used to be more, we used to model it with very specific models and specific statistics in the past. And now, of course, it's moved to this kind of neural network processing. But many of the algorithms inside it are the ones that were developed before. So it's not entirely a new thing that's come now. Most of the algorithms for the inference and learning that you see in LLMs were developed as part of a broader area of machine learning and pattern recognition.
Speaker 2Right, right. Fascinating. And your co-director at Deakin University of the Applied Artificial Intelligence Initiative, could you just briefly, in a sentence or two, describe what they do?
Speaker 3Yeah. So I manage a number of machine learning and AI specialists. And there's also software engineers in our Berwood team. And essentially what we do is we invent new solutions to problems that haven't been solved. And we develop new solutions to problems that haven't been solved. And we develop new solutions and try and look at unsolved problems and try and find solutions. And then we do, of course, translatory work by taking those solutions to solve some problem that somebody has.
Speaker 1And what's an example of a problem that hasn't been solved that you might be working on?
Speaker 3So one of them is assurance. Normally, when you buy a piece of hardware, so say you go and buy a tank or something, you can specify this, whatever, what you want in the tank. And when the tank is delivered, you can just check it off against the specifications, right? But machine learning is a very important part of it. And I think it's a very important part of it. And I think machine learning algorithms aren't like that. So you can get an algorithm from some third party window. And then when you get that particular software, it's been trained on some data, that data may be similar to yours or dissimilar to yours. So you really don't know, for example, if you had a vision system that was trained to work in snow and fog in the US, and you bring it to the Australian desert, and you have complete looks completely different, you're going to test that it's going to work. So assurance is an area that basically before you send your drone out, can tell you that, look, it's going to fail if you send it in these kind of conditions when there's so much specularity or whatever it is that you're looking at. So that would be an example. But there are many, many such problems that are yet to be solved.
Speaker 1Yeah, we'll talk a bit more about assurance in a moment. But what was it about pattern recognition and solving these problems that have not been solved before that really got you when you were younger? You know, what got you here?
Speaker 3Well, it's always been very interesting to me, right, as to what are these patterns. For some of my very, very early work, I was working with this bus company that was setting up cameras for buses in Perth. The company was in Perth, but the buses were in London. And then the London bombing happened. And the single question the police asked is, did the bomber get off the bus? And actually, it could have been answered, because there was a camera outside the bus. So if you had like five or six buses, and you know, London, there's like at least two of them that you see at any point in time, many, many cameras that are looking at that public space, right? That kind of spawned my interest in, you know, the whole area of security. And we thought a lot of the crime happens within 10 kilometers or five kilometers of a bus route, right? So if you think of the bus camera as the roving camera, you can put a virtual camera anywhere. You can say at two o'clock on this location, give me all the footage from all the buses that went there, and we call that the virtual observer. And from that, we went on to other problems in security. For example, I went to the station in Perth, and they had this control room, and they had like hundreds and hundreds of real cameras around there, eight or nine terminals where people were just watching those cameras. So I remember asking that operator how he knew which camera to watch. And he looked at me most doesn't know. So we solved that problem, which is given many, many, many sensors, how do you know a camera has to be looked at? now. And that became a startup company called iSatana. It's actually now listed on the ASX. Wow. So this is just the journey of, for example, how I got in.
Speaker 2And in terms of patterns with that example, is it that the software is looking for anomalies, you know, sort of different types of actions that go outside the normal pattern? Is that what it's happening?
Speaker 3So what we did is instead of focusing on abnormal, we focused on normal. So we learned normal. We found a way to learn the normal pattern. And we said, anything outside that is an anomaly.
Speaker 1Yeah, that makes complete sense. Yeah. You know, there must be so many problems that haven't been solved. How do you work out which ones you want to put your time and effort towards?
Speaker 3A little bit is dependent upon the partners that we have. And sometimes it's driven by need. So for example, we had a person in our lab in Perth whose child was diagnosed with autism. And we were watching that struggle. That parent was having to find a place where that child could be given some early intervention, right? It was almost impossible. The waiting lists were so long. And we were looking at what needs to be done. And we said, look, I think we can automate this. So we then built a solution for that, which was called Tobii. It allowed parents to do early intervention at home. And the program was developed in partnership with speech therapists and psychologists. So it's the partners with whom we work that, in a sense, we're able to do early intervention. And we're able to, in a sense, guide the fact that there is a real-world problem and there's something to be solved. So, I mean, it's about just stepping outside your door and seeing what's a problem. We try to find that. So you can see now that there's all these algorithms everywhere. Everyone's using AI. Does anyone know? You know, I'm really sure, right? So you can see there's some kind of problem. That's how we find the problem, by just actually stepping out and
Speaker 2looking around. I'm curious, how much has the game changed since generative AI? Burst onto the scene a couple of years ago. In terms of what's possible in your field, has it made a big difference?
Speaker 3Yes. I mean, we were moving towards this, right? And I never thought it would come in my lifetime. So it was quite fascinating, even to me, that it happened. I think it has changed everything in very profound ways. And I don't think we've quite come to grips with how it has changed the world in so many ways, right? Like, for example, if you take education, right? The point of education was to teach us, at least, for example, in science, right? They would teach you facts of certain types and how to logically reason through those facts, right? It doesn't seem to be important anymore to know the facts, right? So what you need to know is perhaps some way of inference or some way of being able to synthesize information across diverse fields rather than this one small field. But because I could only learn the facts of this one small field, right? So you can see that what we teach has to be changed, right? Similarly with research. So we would do some, for example, in research, we would do something like, okay, let's invent a new material which has these properties, right? We would take three years to do it, right? And now you can pretty much do the same thing with the assistance of an LLM in a far shorter time, right? So then the question is, what is the value add that we're going to add in this research? And so you can see it's not the same direction. It's not like, I'll do this faster. That's not a thing, I think. I think what you have to do is do some synthesis and get some broader insights across all kinds of disciplines to be able to do a rich piece of work now, right? And that's where I think the human ingenuity comes in. So it's completely changed, I would say, in terms of not just my area, like all areas have changed.
Speaker 2Presumably the generative AI, you know, context windows and the like, allow us or help us to bring those different disciplines of knowledge together too, because again, they can help you at least absorb and get an initial sort of digestion of cross-pollination of ideas and things like that. Would that be the case?
Speaker 3Well, we're not trained like that. We've been trained to think in a very specialized kind of way. As researchers, we are very specialized because it's very hard and very deep. And so we don't have the ability, I would say right now, easily. In fact, I'm trying to do this in our institute right now, is have a new kind of seminar series. You learn about the latest work in some field, of course, but I do a section called Connector, which is from 1950 onwards to now, right? Like what happened? What happened in neuroscience? What happened in biology? What happened with living systems? So it's to try and make the students change the way they look at a problem, which is very diverse. Although you can ask the LLM, you don't have to ask the LLM, you don't have to ask the LLM. You don't know what to ask. If you have a broad view of everything and how to think about
Speaker 1it. Yeah, I think that's the real challenge, isn't it? It's actually knowing what's the right question to ask. Exactly. And it comes from
Speaker 3your own ability to do the synthesis on your own without the LLM, right? That's the skill we have to teach them.
Speaker 1Yeah, I think that's the same in business as well. I think people are going to have to have those kinds of skills because they're not going to be, you know, vertical specialists anymore.
Speaker 2Exactly. I'd love to come back to, you touched on AI assurance and you gave us a tank example. I'd love to try and translate that into a business context because, you know, what we're hearing, especially the smaller, more agile, closer to the frontier companies are already sort of letting loose AI agents. Am I right in sort of saying that in that kind of context, AI assurance is what you would have to make sure that those agents, stay within the guardrails and the brief and the goal that has been set them versus over time, have a mission creep? Is that a fair summary of the sort of thing you're working on? Is that a relevant context for what you're working on?
Speaker 3It's like two things. One is the question of accuracy itself, right? So you take, for example, some companies that do medical scribes. So you go to the doctor, the doctor says, I'm using some AI to do it. How true is that transcript? The doctor is not certifying it, right? So there has to be another thing that certifies that this transcript it's got from this conversation is medically correct. The example that one person gave about being with the doctor and then the doctor got a phone call and started transcribing the phone call.
Speaker 1But it's funny because I was, I was a specialist and he was using this transcription technology. And he said, I have to say this word because every time AI gets it wrong. So I have to actually spell it to the AI.
Speaker 3Right. That's a very vigilant doctor, right? So, but you can imagine that when they're having lots and lots of patients, they're not checking each thing that it's doing. In some sense, the owner shouldn't be on the doctor. It should be something that can certify that this product is working as it is. So there's the accuracy issue. The issue of mission creep is even more scary. So if you can imagine these agentic systems, say I built an agentic system and I'd give it access to three sites. And one is to say, okay, it's only going to those three sites. It's not going away from those three sites. That may be easy to handle, but you can imagine there's this prompt poisoning thing that can happen because it can come back and poison, change your preferences. I mean, it can shift the way the LLM talks to you in a way, right? And monitoring all this, following all this, these are open problems. And what happens like in the movie 2000, if you remember what, there was only one Hal, but if you have like 15 Hals, we're just as good as, right? So you can imagine these 15 Hals colluding. I mean, we only needed one Hal to get out of the spaceship, right? But now you can imagine how complicated that is to find out whether these agents are colluding, whether there's something going on. I mean, all that's open.
Speaker 2Yeah. And so the ideal presumably would be that the assurance is an ongoing continual sort of in the background monitoring. Is that sort of correct? It has to be because the systems are learning.
Speaker 3So see that, that's all the extra jobs that'll come. Yes. They'll come because of the speed at which these things change. And then somebody has to monitor them. Every company will have their own little group that does that. And so a lot of the jobs will go in, in these kinds of assurance type activities, I think.
Speaker 1Yeah. And do you see the technology then working alongside the humans sort of augmenting, the humans' ability to be able to do the assurance?
Speaker 3Yes. I'm not clear exactly how right now, but it should be working simply because the human doesn't have the capacity to audit it. That's why we're using the algorithm, right? So for example, suppose you have an agent that is talking to children, and this is one of the projects we're doing. We want to make sure it's safe. So the algorithm we built basically finds the boundaries of the space in which the LLM will start to work. And so the algorithm we built basically finds the boundaries of the space in which the LLM will start behaving in strange ways, right? So now the question is what the human will do with that answer. That answer is not a simple answer. It's not like in one paragraph. It'll be like, it'll give you some space and then you'll have to synthesize it. You'll have to figure out what to do with it. So there's a huge human role in this assurance process as well.
Speaker 1Yeah. I think that's actually great to hear. Some people have got a view that it would be sort of AI on AI, if that makes sense. Yeah. I sort of agree.
Speaker 2have, because, you know, I'm thinking of deep mind and cybersecurity, where a bit like your video software, which picked up the anomalies, I understand that that's kind of a key cybersecurity tell, is to monitor and look for unusual behavior. And then that attracts the attention, flags it to the human who then steps in. Do you imagine it's that kind of partnership between AI and humans with assurance?
Speaker 3I'm not sure. Because the more complex it gets, the human gets more out of the loop.
Speaker 2Yeah. And overwhelmed with the scale.
Speaker 3But I think we have to build systems where the human is somehow in the loop. Yes. We shouldn't be in a position, if you remember the 2000, I think that's such a good movie. Because if you remember, they ask Hal to turn Hal off as if that's going to happen.
Speaker 1Yeah. There's no way Hal's turning Hal off.
Speaker 3No way, right? Yeah. And that was a movie. But you can see they're going in that direction. Look at another agent to stop the other agent. It's never going to happen. I think that movie is interesting from the perspective of we're there now. You need to be able to see. And I know it projected a dystopian future, but that dystopian future is within what can happen as well, right? Yeah. It's not necessarily the only future, but it is a possibility and one must prepare for it.
Speaker 1Yeah. And it's interesting because you say we're there now. I mean, obviously, as Greta said, there are a lot of companies starting to implement agentic AI and swarms of agents. But it feels to me, by what you're saying, that assurance is actually quite a long way behind. Is that right? It's very behind.
Speaker 3There is not a single algorithm now that can do these absolutely safe border checks of many agents with all of this stuff. Can I find out if it's colluding? And that's one type of bad behavior, right? There would be so many types of bad behaviors that agents can do. And do we have a system that can. Can detect it, sort it out right now?
Speaker 1No. That's actually quite scary.
Speaker 3We have some solutions, like you'll find a paper on collusion detection, but that's just a paper, right? Like it's not like a system out there.
Speaker 1Yeah. And how far away do you think that is?
Speaker 3I mean, I think it's at least a year.
Speaker 2So what advice do you have for listeners in the interim, particularly, I suppose, at that organizational sort of level, but also as an individual? How do you use AI safely given the lack of assurance and other safeguards at the moment?
Speaker 3You can use it in many, many things for which it should be fine, right? Like, for example, if you're in a process plant and you want to find anomalies in your process data, or you wanted to predict when the next fault is going to occur, or when you wanted to predict the next. These are classic machine learning problems, right? All those ones you can do safely now. It's only when you get into this kind of territory where you give over your email to the agent and you. I mean, if you give access to everything and then it's something we've never done, right? And don't even give it another person complete access to all our emails and calendars and everything, right? Now, suddenly that is possible to an agent. And I think companies will work so that they say, these things are okay and these are not, depending on where they see the risk.
Speaker 2Sveta, you've had decades immersed in AI and machine learning. I'm really curious, you know, from your perspective, if we think about the workplace, people are very worried about job security and jobs disappearing. How do you think about what jobs will go and what jobs or tasks will stay?
Speaker 3I mean, of course, nobody knows exactly where all this is going to go, but I feel that in the future, the jobs that have very precise definition. So if, for example, you've got. You've done your requirements engineering, you have very clear requirements of what software needs to be written, then a machine can do that. So if there are jobs like that, where it's very structured, very logical, and it's very clear as to what needs to be done, those jobs might be reduced. But if you take jobs in which there is a strong human component, where it's not so structured, and then the human has to use judgment to see how things flow, then those jobs can't go. Like even, for example, your jobs as podcasters, you can't give a script. For, you know, there's this person, this CV, run an interview. You can, but it'll be very robotic and not in the way that you do it. Similarly, for example, when you go to a GP, maybe the components of partial components of diagnosis can get automated, but the component in which they walk you through the risk, take you through the uncertainty, help you navigate, those are things that cannot be taken away and therefore will always be there. And there will be these jobs in almost every organization. If you analyze your organization, you'll find there's always components where there is a human element that's required. And those jobs, I don't think will go away.
Speaker 1Just, you know, think about in your day-to-day life, and imagine you're a much more proficient AI user than the majority of people. Where are you finding the great benefit personally from AI in your work, your personal life, et cetera?
Speaker 3So I think you have to know what you want to do. If you know what you want, if you know what you want is fantastic because then shortens the time to get from where you are to where you want to be. If you don't know what you want, then I find it very disturbing because it'll try and take you in directions and get rabbit holes and stuff like that. So I think it's really important that the part of thinking that you used to do before you still have to do, you need to have a very clear idea of what you want from the system and use it as a tool is what I said. Use it as a tool. I mean, some people say it's a partner and all that, I think, yeah, you can, like for brainstorming or something like that, you can use it as a partner, but in general, the advantage I have found is thinking of it as a tool to solve something that I don't want to do, but it can do it for me.
Speaker 1Yeah. What would be an example where you found the greatest added value from it?
Speaker 3So for example, in the seminar series that I was talking about where you have to do the connector section, we took the subject of memory, right? Like how do biological systems have memory and how do computer systems have memory? In computer systems, I have a very clear idea of how memory systems have developed over time. So I ask it to say, can you summarize the seminal papers? It'll actually do that quite correctly and I can check it. And similarly, can you do the seminal papers in the biological systems and it'll do it. And that's really useful because this would have probably taken me a week
Speaker 1otherwise. Yeah. That's a great case study, actually. Yeah. Yeah. If we take a step up onto the balcony and we think about the next sort of three to five years, what are you most worried about when it comes to AI?
Speaker 3I'm worried at things at the societal level. I feel the form of universities will and has to change in some way, quite fundamentally. I think there's inertia in such systems. So maybe three to five years is not the right time. I don't know what the timescale is, but eventually it'll have to change. I'm also worried about the amount of time that we're going to have to spend on AI. So I'm worried about the amount of time that we're going to have to spend on AI. Like, suppose you were, if you had a child who was finishing school, what would you advise that child? What skills would you advise that child to have? And how would you train that child to be ready for the world? And if I think about it, I think those skills are very broad. It's about how to deal with uncertainty, how to synthesize information from incomplete information across And the doctor gives you these choices. You should have the capacity, not with shared GPT, but to reason through those options yourself, right? And if you have this option for your, some problem that's confronting you in such a personal way, you will be able to solve any problem at your work in the same way. So it's unclear to me exactly what these jobs will look like and whether there'll be as many jobs. I mean, there's two points of view on this. One is that there will be an equal number of jobs. There will be an equal number of jobs. There will be an equal number of jobs created as with every other technology. And the other job is that, as Elon Musk's view, is that if everything becomes so cheap to produce, there is like no value in it, right? Like there's no value for humans. If the machines do everything, then what exactly do humans do, right? So that's the other point of view. I'm worried about things like that because I don't think any of us have clarity on that. And I think these are the issues. And what will happen to society? Will there be some people who can really use AI and know what's happening and then some people are just consuming it like social media? And then that's even, this is a much worse form of how it can influence them. So that's what I'm most worried about.
Speaker 1Yeah, yeah, yeah. I absolutely agree. And the same question, but what excites you the most?
Speaker 3Well, I think to me, it's exciting that it's now an opportunity to build a new kind of university. I don't know whether I'm the right person to do it, but I would like to think that I can at least think through that vision, right? Because it's different from the Humboldt model that was done from the 1850s. Nobody would just change, right? But this is, I think, the impetus for change. I think it's a good thing. I think these colleges, maybe it becomes more like a liberal arts type college that you have where you learn a broad range of disciplines. And I know they don't have sciences, but sciences have to be part of that as well. And that's the opportunity. So the opportunity is to become a much more interesting, cured. We've kind of a little bit become like ants, right? Like we go into one specialization, then we do that same same specializations for the first life. Well, maybe we should be more broad. We can think more broadly. We can contribute to the community. We can become broader people, I think, through this
Speaker 1opportunity. That's fascinating. And it's interesting, you're thinking about universities, obviously, because you're sitting in one and that's the environment you're in. But I think it's a similar question for all organizations.
Speaker 3Yes. In fact, one of the CEOs of a very big law firm called me not long ago and he asked and he wanted to brainstorm with me what his organization would look like in five years. He now has 800 lawyers, but he can clearly see that he's not going to have 800 lawyers. Right. So what does that mix of talent look like? How will they work together? I think these are the questions that businesses should confront right now.
Speaker 1Yeah. And it's very difficult to sort of imagine what that might look like. But I think you're right. They are certainly the questions that you should be looking at various different scenarios of. Yeah, because it's so hard to give any distinctive, clear answer.
Speaker 2It really is. Scenarios make sense and then you plan. I'm interested. What was your advice to him?
Speaker 3I didn't have any, but I think he himself had very good ideas. He was basically saying he wants to employ people who are just creative, right? Different types of people. And I think his answer also seemed to imply that people who were trained to look at problems in general will know how to navigate. Whereas if you were just a lawyer and you just know that one case, Chachibidi can literally do it, right? So you want to be broader than that.
Speaker 1Yeah. That's sort of music to my ears because both Greta and I have had these careers that are non-traditional that go across lots of different areas and that hasn't been normal. You would be the exemplar. I don't know about that, but certainly I think the diversity of experience and being able to problem solve in different arenas is important. So it's been a really fascinating conversation, Sveta. We've loved it. If our listeners wanted to find out more about you, more about the work that you're doing at Deakin, where would they go? You've got my email, right? Great. Well, so we'll put your email onto the show notes. So if anybody wants to connect. So, well, thank you so much for joining us, Sveta. We love the work you're doing. Please keep going on assurance. We so need it. And it's just been great to talk to you. Yes. Thank you so much. It's been a pleasure to talk to you
Speaker 3both. Thank you.
PodcastTranscript AI