Testing AI Models as Magic Boxes: Why Checks Fail - Michael Bolton

We're ready for this, this is our moment, right? This is our time in the sunshine and all that old silly stuff about reducing testing to a checks about put from functions. That stuff doesn't have a chance against AI. Machine learning models are not developed by a conscious, intentional human being. They're more like selected from dishes of bacteria, petri dishes. We can't write a chapter on AI because we know that it's going to be at a date for the Incas Dry, right? That's just the way things go in this silly business. We're being paid to learn stuff on behalf of other people. What's a better job than that? Welcome to software testing and leash the podcast for people who want to build great software. And also a warm welcome to the Eurostar community because this episode was recorded at the Eurostar 2026 in Oslo. I hope you enjoyed the conference as much as I did and maybe we will see each other at Eurostar 2027 in Copenhagen. My guest was the one and only Michael Boat. We talked about testing AI as a magic box. Why you should not depend on magic when something really matters. Why simple checks are hopeless with language models and how these models fool us like a mentalist on stage. This and more you hear in this episode and now enjoy. End one note before we start. Are you moving your test automation to a new tool? Then you know the problem. Migrating thousands of old test cases one to one costs time and a lot of money. And often this is not necessary. Team scale shows a lean-away in a free playbook. You migrate only the most valuable 20% of your test and your bug detection stays almost the same. You will find the playbook under tul.fm/migration and the link is also in the show notes. And now enjoy the episode. Hi, Michael. It's great that you are here on the show. I'm delighted to be here. Yeah, yeah. The final day on Eurostar. Yes, nearly the end. It's fading out. It was just a real next year in Copenhagen. I hope to be there too. Yeah, yeah. I think you will be. You never know. It's great that you are here in the show now. And I saw your tutorial, not in person, but in the abstract because I was in the program committee. And it was about testing the magic. Testing magic box. Yeah, yeah. And I want to elaborate a little bit what's behind that for you. It's starting when we were writing our book. Taking testing seriously, the rapid software testing approach. Which we've been writing for the last 30 years. Yeah, that's how long it takes to write a book. But also really seriously for the last five years, we get right close to the end of the process of writing the book. And we've got to write a chapter on AI. Except we can't write a chapter on AI because we know that it's going to be out of date before the ink is dry. So we wrote a chapter on magic because we wanted the chapter to be future proof to some degree, right? So that prompted us to think about how we think about magic. AI being a magic box. So first, the zeroth principle here is that a box is magical to the degree. Some boxes are strongly magical. Some boxes are weakly magical. A box is magical to the degree that its behavior is neither controlled nor understood nor otherwise known to be safe. Yeah. Okay. Now what does that mean? That means, for example, the Python interpreter is a magic box. To me, because I don't know how it works. I mean, philosophically, conceptually, I know how it works. But I don't know the details of how it works. So in that sense, it's fairly strongly magical to me. Controlled. Well, I guess I can control its behavior to some degree. So that makes it a little less strongly magical. Otherwise known to be safe. Well, in that sense, although I don't control it or understand it completely that thoroughly, there's a worldwide community of Python users out there. And they're also people for whom the internals of it, they do understand it and they do control it. So we have social proof that it's safe. And we have some kind of belief that if it were fundamentally radically unsafe, then the people responsible for curating and maintaining it would fix it. And they do. I mean, from time to time, that's what happens. We find out ways in which the magic isn't working or it's, you know, we try to pull the rabbit out of our hat. And it turns out to be a rat. So what we're dealing with at the moment is that AI in the broadest sense, mostly based on things like machine learning models. These things are constructed without a human specifying the construction of them. So it's unlike software, regular software in that way. And we see one that kills off a certain kind of nasty bacteria. And we say that one looks like the best so we pull it out. Yeah. And this is one we want. Okay. That poses special kinds of problems and risks because we don't know exactly how it works. It's like trying to figure out what good antibiotics were before we had the capacity to do genetic analysis, DNA analysis and all these things. So now we can do it with proteomics and genomics. We can sort of get a sense of the mechanisms that are driving a particular antibiotic. And now with CRISPR, we can edit it. Yeah. That's really cool. So okay. All that since these things are strongly magical boxes, nobody knows how they work. Nobody has a source code for them. Even if we did have the source code, there's a certain amount of stochasticness. Stochicit. There's no hope in me being illicit. Stochasticity. There you go. When you're out having fun with Finnish people the night before, certain things get hard to say. So anyway, these are strongly magical boxes, which leads to a set of principles. Number one, if something matters very much, you probably shouldn't depend on magic to get it. Yeah. All right. Don't rely on magic to obtain it. If something matters very much and you actually have no other way of obtaining it, well, you might as well try magic, but it behooves you to minimize your reliance on magic. You don't want to depend on it. All right. If something matters very much, this is number three, if something matters very much, and you are using magic to get it, responsibility demands that you test it. Yeah. If it matters to yourself or to other people, especially if you have a duty of care to those people, as we do, people in the software business, we have a duty of care to our clients and to their clients, their customers. So we must test it. If we are going to test it, we want to maximize testability. Now, there's a lot of testability in the traditional sense that is unavailable to us, because once again, we don't have the source code. We can't instrument the source code itself to make it more testable, but there are some things we can do to monitor inputs and outputs closely to understand the training data that went into these things, to understand the guardrails that are around them on both sides, to understand factors that might make a difference like the context window and its size and what's in it and what got retrieved and how we're accessing databases, how we're making calls to services in the outside world. All those things, if we're aware of those and we can control those and observe those and understand the behavior of those things, Then we're reducing the magic nest of a magic box to some degree. So part of it is There's a GPT in the middle of a magic box. Maybe we can Put little windows in the bigger box or maybe we can put probes in it or little cameras. Yeah, so all that stuff So that's the fourth principle maximize testability and Finally, if you can't test it or you can't test it thoroughly enough Then there are other things you've got to do you've got a disclaim it But even if you disclaim it and if you are foisting it on other people you have to accept responsibility for that So we're into some pretty interesting areas for testing and for responsibility and ethics and For thinking about how we go about the business of understanding complicated stuff. Hmm fortunately The kind of testing that we've always talked about is ready for this. Yeah Because the kind of testing we've always talked about so testing is an investigation With a special focus on looking for problems testing for us is evaluating a product by learning about it through experiencing and exploring and experimenting. Yeah, yeah We're ready for this. Yeah, this is our moment here at this is our time in the sunshine and all that old silly stuff about reducing testing to Checks about put from functions That stuff doesn't have a chance against AI. Yeah, it's a useful thing mostly for developers to do With deterministic software that they understand they can instrument When it comes down to LLM's It's hopeless. Yeah Now we can apply that kind of work to some degree Usefully to all the bits of An AI-based system that are good old-fashioned software as usual We can't Apply to the LLM itself. Yeah, they're there are certain things that are checkable within a big LLM system It's interesting what what comes to my mind is that some clients of me they have also this way going away from checking stuff with With this test cases and now when the AI is coming they rethink what are my risks? What quality do I want this old questions? We should have asked a long time. They are now getting aware of that We've been asking those questions forever. Right right right is it what what is? What is important? Yeah What do you need dear client? To help you understand how to run your business and how how can I help you with that? Yeah Yeah, what are what are the things that are going to threaten the value of the product to people who matter? Yeah to you and to the people who matter to you. Yeah, yeah Yeah, that's it's I think it's it's it's much more now about this conversation about this this stuff and to get this Information out of that and then then we can deal with it in any way. Yeah, plus I got to add I've been doing testing of Language models recently and two things make it fun It's iterative and spiraling in it's like or another way put it. It's like peeling an onion. Yeah The more you peel the more there seems to be there there. Oh, there's another layer down here and then another layer down here It's like from that line from pure gint When he's opening up the onion. Oh, there's another layer here. Yeah, and then he says nature is witty. Yeah Yeah, which is really cool So what I'm seeing is patterns of inconsistency and unreliability in these things that among other things it's been occurring to me Maybe this is at the bottom of what's going on when people say Cloud code used to be good and lately it looks like it's kind of sucking well I'm seeing behavior at the very lowest levels and acquiring the model directly. Yeah, I'm seeing behaviors that don't exactly Explain why that's happening or how it's happening But boy, can I ever see that play out the differences between 3.5 3.5 Haiku 3.5 and Haiku 4.5 Are really really interesting 4.5 tends to go off the rails in the experience that we've been doing a lot more quickly Yeah, and a lot more bizarrely then 3.5 did but the other thing is they all do it Yeah And I am on the verge of some people in the class saw this. I'm in the verge of publishing Some of the observations of it and the obstacle to it is There's so many and they're so weird. Yeah, yeah, and I want to do a really good job of this But the other the nice thing about it is we've got an experimental framework now that allows us to run these kinds of experiments over and over again and see Patterns and see weird outliers and The refinement of the experiment allows us to See things in different ways and to look for different things and have software help us pick those things up That's what's a fun - yeah, yeah, so I'm I'm I'm having a good time with that kind of stuff and it's fun Yeah, testing is always been fun if you do it right if it's not that's what if you're not doing it right That's when it's not fun That's what I was I laugh on my job. So since 30 years now We're being paid to learn stuff on behalf of other people. Yeah, that's true. Yeah, yeah, so so you were dealing with a lot of AI stuff now testing this and and exploring this and and What do you think is what is the the problem for the most testers? They are complaining at talks here or at Conversations that it's not testable. So what this what what what what shift is needed for them to oh, man a few things it keeps going back to this Thing that we have been refining for many years and now we're at the point where we think we got it testing is So I said evaluating a product by learning about it through experiencing exploring and experimenting now That makes anything Testable. Yeah, we can always experience something because I was get experienced with it we can always Evaluate it by learning about it. Mm-hmm. We can always perform experiments with it and That process is fundamentally an exploratory process because We need to be in control of it right there is no preset predetermined way of Learning about something that you don't know about yet and that to agree nobody knows about right so That's fun That is the excitement about idea and an observation and a procedure that that allows you to enact it Prompting some new discovery. Yeah, and when you see a new discovery that prompts a new idea a new hypothesis a new procedure for experimenting or something or a way of going deeper on the experimental process you've got For what you have a new patterns new new observations of things like bias and behavior and Also a recognition that the least little thing Can cause no change whatsoever or completely bizarre new thing. Yeah, and it's exciting. It's fun to do that Yeah, I think it's a great perspective because when when we see there's this quote that magic box or magic Technology is magic as long as we understand it then and then the magic disappears and this repeats every Decades in the in the past and I think we as a test us when we are going Exploring the stuff and then the magic maybe a little bit disappears, but we have fun on this process well, you know, I'm a big fan of magic I whenever there's an opportunity to see a magic shot to take it and One year there was a test in conference in Las Vegas So I went to see pen and teller now pen and teller are my favorite magicians because So often they will tell you exactly how a trick is done. Yeah, okay They they do a wonderful routine with the cups and balls where they show you Exactly how that that's the one where you know there's a cover a three cups and then Up to three balls, I think and they all move around and they all they all seem to appear and disappear and so on Well The coolest thing about Their delivery of that trick is they do it with clear plastic cups. Yeah So you get to see exactly how it's being done, but as a consequence of that, you get to see how clever magicians are, and you get to see how easy it is for us to be fooled. You get to see the specific mechanisms of how we're fooled, and that defends you against being fooled from being fooled in circumstances where there's another trick that they're doing. What it means is you can start thinking in terms of human ingenuity in terms of what appears to be magical, and that is a deeply humanizing kind of thing. So the same with a particular piece of technology. If I understand how technologies like it work, then I can understand, for instance, with a large language model, I can understand the basic principles behind its operation, which is basically to string one dang token after another. But I can also, from a psychological perspective, because you look at that too, I can look at how patterns of text fool us into believing that there's insight here, when there isn't. Such insight that is there comes from us. So it's not the LLM that's saying anything interesting or profound. It is our effort to interpret it and to make sense of it that is causing neurons to fire in our heads and helping us to think of things that we had not thought of before. But what that in terms allows us to do is it allows us to give credit to ourselves, think of credit to humans, instead of giving credit to the machine. There's another wonderful essay which I try to recommend to people, three or four of them. One is Stephen Wolfram's description, what is chat GPT doing and why does it work? That explains the mechanistic aspects of the GPT's process, very cool. But why does it work and how does it work? Has another dimension, that's a psychological dimension and that's really beautifully covered in an essay by a guy named Baldur Bjarnason. The title of that essay is the large language mentalist effect, something like how large language models replicate the mechanisms of the psychics con. And how does that work when people go see a cold reader, tard reader, palm reader, tarot card reader, mentalist. By the way, I just have to say, thanks Alan Richardson, who is on the program committee that year. In 2013, one of the people on the podium, when I was the program chair, was a mentalist. So in 2013, we were ready for, you know, we're preparing to deal with this moment. But anyway, how they work is they say something quickly looking for a hit. I give this example in classes sometimes. Now I'd like you to think back to last week. Last week was a fairly, well, this is a little busy for you, of course you're preparing for a year or so, and there was a technological complication at some point. I'm thinking Wednesday or Thursday, there was some piece of gear or some piece of software that you had that frustrated you for probably like half an hour yet to go back and fix this thing. You're remembering that, right? And it was a piece of productivity software now, it doesn't matter whether or not what's going on in your head right now is you're trying to stick that on to an event. And that is the magic of GPT's as Baldur points out in this. I was really, I was annoyed by his essay. Here's why. I wanted to do that essay. That's one of the reasons I was annoyed. The other reason I was annoyed is he did such a good job. I looked and I said, I could never do something I could. So, I recommend to testers who are working with these things understand how they work mechanistically, understand the effect they have on people's minds, and that defends us against being fooled by them, bring that to our clients, and then say, all right, we've got to figure out whether this thing is doing a good job, whether it is satisfactorily going to provide us with consistent or reliable results. Sometimes reliability doesn't matter. If the job for a GPT is to write a funny poem about a coworker for the Christmas party, no problem, no risk, because it produces one that you don't like, you can throw it out. If your bond is giving medical advice, big risk, so maybe we want to be cautious about what we let the bond say and not say. Maybe it might be a good idea just to avoid using magical technology at all in places where the risk is elevated, where things are on the line when something matters very much. And that's a principle that are so true for years, but now much more important for them. Absolutely. So understand the technology, engage with it, encounter it, use tools. We have to do so much statistical analysis of these things that anything other than a total first hurdle test, we're going to have to use tools to generate large amounts of requests and responses, but we can use tools to help us induce variation into the request. A great application for an LLM, ironically, is to have it generate zillions of variations on the same question that we can ask from LLM and watch to see how that changes things. Because the introduction of a single word or the change of a single word can dramatically affect the output, what we want to see is how dramatically and how often, and to what the green might just completely invert the meaning of something that people might be asking about. Yeah. So great insights. Thank you very much, Michael. I could talk for you for hours now, but I think we have to continue this on another episode too. I would absolutely love that. And anytime. Yeah. And I think you gave so many tips here for testers. I think we have to reflect about that, read the book, maybe get into the paper, rethink what this magic and how can we deal with this problems, the risks of that, yeah. Magic is fun. Yeah. Magic is fun. Magic is fun. Magic is fun. This is one topic for this episode. Yeah. Thank you so much that you were here on the show. My pleasure. Thank you. Thank you. [Music]