AI-Native Engineering Culture from a CTO

The big milestones for us were we started from everybody's using AI to code. We moved into, "Hey, what's the last time any of y'all actually looked at the code?" And all my team was like, "Like, everybody didn't want to get caught." I was like, "Oh, because I haven't looked at it in months." Yeah, and they were like, "Oh, god, okay, thank you, thank you." And Poster Syndrome that jumps out when the juniorist member of the team posts something in the engineering Slack that calls out something the most advanced engineers done. And you're like, "Oh, god, I'm not even a real engineer." And then you're like, "Oh, wait, you had caught a look at that, didn't you?" And they're like, "Actually, no, it was Codex, you know?" Welcome back to the Agentic DevOps podcast. And I'm your host, Brett Fisher. I have my friend, Mike Rollins, on. We went to lunch, and the way he talked about his AI first engineering, or agentic engineering, or agentic development, whatever you want to call this thing, where you're throwing tokens mostly and not writing anything anymore, and you're not even looking at code anymore. And he runs a team as a CTO, a small team, and a startup. And I wanted to talk about what that life is like. Because a lot of us, you know, we're in enterprise, or we're, I would call like, legacy teams, where we're towing around with AI. We're certainly using it for code, but we're not necessarily riddling the entire pipeline of the software life cycle with everything AI, so that we never see code again, and that everything just goes as fast as possible. Like, most of us are somewhere in the middle, or near the beginning, I think, and certainly with a lot of the people I talk to, where they aren't necessarily able to throw out the playbook that we've had for the last 20 years on how exactly a team uses Git storage and automated deployments and test driven development, and all the things that maybe were hot topics before AI, right? So Mike and I go through a pretty lengthy conversation, but I wanted to get as detailed as possible to talk about the culture, the pros and cons. Like, what are the downsides of this? Tell me some horror stories of how this can go wrong in so many ways, and what it's like trying to push the envelope being on the bleeding edge of AI first development. So let's get into it. Mike, welcome to the show. Tell the audience who you are and what you do for a living. So, my name is Mike Rollins. The majority of my career has been in Silicon Valley. Back in '22, our company was getting bought by Twitter, and then this dude named Elon jumps in, and we were no longer being bought by Twitter. And as a result, I went on this multi-year escapade of I worked for a healthcare startup, and then after the health, I left the healthcare startup, or I was asked to leave the healthcare startup, because it turns out if you ask very difficult questions pointedly, people don't like it. I ended up going out on my own, and it was terrifying, scary. I mean, from about 2023 until really November of last year, I was a contractor, and I did a bunch of stuff. I was the founding engineer for multiple projects. I've been the CTO now of three different companies, though mostly fractional. And I have, at some point, the internal clock in my head is always ticking, and I feel this constant need to go faster. And so, as I was building things, as typically the only engineer, the clock in my head is just faster, faster, faster. And so, when AI became like a thing, meaning back in the latter half of 2024, and then especially at the beginning of 2025, myself and a few other folks, Kenny and Thanas, who he and I have a podcast together, they were kicking back off, called Willa, I snap that. We both just dove in. And what's funny is that since about really the entry of Cloud Code into the market, I have been doing it first engineering. And what I had noticed is that by the time we hit about Mayer's June of last year, I was no longer writing code. And then right around November of this year, I realized I was no longer reading code. And it was this shocking realization. And so, I basically dove head first into AI. And, you know, high risk, high reward, the risk here is that my family can't eat, right? The reward is that I figure out a way to do engineering that other people don't know. And that's what we did. And so, nowadays, I'm at this company called Relify, and then also Family Digital, I'm CTL of both. And I have this team of four other engineers, and really truthfully, only one of them. Well, one of them is a designer that we've turned into a front end engineer. And none of us read code, none of us write code. We ship, we've built this agentic system that's built from the ground up for sharing. And it is kind of, it's built at right angles to the way everybody else is doing things. And we are moving so fast, it's terrifying. But the clock continues to tick. And I continue to think, how do we go faster, faster, faster, faster. But I do never in my life, I got to ride mobile. And I started riding code for iOS and then Android. And mobile felt amazing. It felt like a freaking rocket ship. Everything changed so fast because there was a new phone about every three months. And now it's good. Didn't we just see a new frontier model a week and a half ago? Because we did. And it's just unbelievable. So anyway, that's how I ended up here. Yeah, that's a great story. And I was on the hardware side of mobile, not the software side. So yeah, it has become very apparent to me in the last few years that it's like, I, it's a good analogy of like AI feels like it's kind of replaced mobile as the exciting tech where that was really, I mean, mobile kind of came around a little bit. A few years after cloud, but cloud, I feel like was pretty well understood within four or five years. And then we had mobile and that was exciting. And then it was like this lull, I kind of feel like 2018 to 2020 was kind of this lull of we were all sort of, yeah, we were all sort of exhibiting the same pattern. The vacuum of crypto, which, you know, I don't know anyone that touches it with a 10 foot pole. So yeah, that was the thing, but not necessarily in my ecosystem. So we all were talking about how isn't it great that Docker and Kubernetes are boring? Isn't it great that like microservices and distributed services and cloud tech are all kind of boring? And we can just focus on the work and enhancing our craft and getting better at it. And then, you know, touch the Pt shows up and then we're all copy pasting. So I wanted to get in some of the nerdy details of it sounds like it's a relatively small team. So total of five, it sounds like there's still a five of you. So does that mean you have five people making PRs into presumably get up? And how do you go about even starting something? Is it like an individual thing where you start an idea? You mentioned this idea of concepts first before ideas or something like that. Have you been adding text-to-speech in your apps? One of my late-night projects this year was creating a personal podcast feed so I could listen to the change log of my favorite open-source projects while I do my chores. I needed a text-to-speech API to do that. I also heard from a colleague this week who built an app for his kids to turn their e-books in the audio books and he's using the Speechify AI developer platform to do that text-to-speech. Speechify AI is a developer platform for real-time text-to-speech and they make their own voice models. Their Simba 3.2 model ranked number one on the artificial analysis benchmark for conversational voice quality. They've got a free plan, streaming, and rest APIs, TypeScript, and Python SDKs and if you want to get fancy, you can use their SSML metadata support to add a motion into the speech output. Whether you're shipping a weekend project or production voice agents at work, you can start free at speechify.ai. The link is in the show notes. Thanks so much to Speechify AI for sponsoring this episode. Well, let me start at the end for you. Okay. Because this is typically the framing, this is the framing I've been aiming for and we're almost there. What does it take for me to allow an agent to pick up something, build it, test it, commit it, open the PR, make sure everything runs, listen to all the feedback, merge it to main and deploy it, and a human is never involved in this matter of fact, a human may never know that it happened. What do you have to do to make that real? Risk is really the big asterisk there. As the clock is ticking in my head, I make decisions based on a risk assessment. What is the likelihood of failure versus what is the outcome if we fail? What you find is that even if the likelihood of failure is high, but the risk associated with the actual failure is low, you may choose to take the gamble. Conversely, if the likelihood of failure is very low, but the risk is extraordinarily high, you may not take that leak. It's not a gamble. It's almost a guarantee, but you know that if you hit the enter on this select statement in the production database and it goes sideways like the failure mode could be the database disappears and then so does the company. That's one of those things where it's like, I've made that decision and had it go bad. I have wiped the company database. Are you the one that's like on the tip of the spear so the rest of us can learn from your failures? When I was at outlier, we did machine learning over time series and this was in 2022. That was the company that was going to get bought by Twitter. I was on call for three years straight and I wrote direct queries in production databases constantly because it was the only way that we could keep the system up and running sometimes and wake up every morning and I would rebuild the system and there were a couple times that boy did I bad out. Even in a couple not the current company, but the previous one, there were a couple times I just I wasn't thinking I knew I was doing something a little bit risky and I did it anyways and I had to go desperately hope that we had a recovery copy and stashed away and we did and so I have legitimately I've blown it all up. I'm very risked tolerant in that aspect, but the reason I'm risked tolerant is I fixed it and so when I look at let me give this as an example, we had an incident occur a few weeks ago and All of a sudden I realized that we could no longer upload files to our system and now keep in mind It's an agentic system. You create a bunch of agents. They can all talk to each other It's a lot like open-claw except we were doing it first and we built it to be enterprise safe And part of that is like you have to upload things and they get shoved into the rag database and they get shoved into The S3 file system and I tried to move a bunch of things around and I couldn't I was like what's going on now I'm at my daughter's dance competition. I don't even have my computer with me I'm the only engineer that can fix this at this moment and so I'm actually on my phone So I swipe over to cloud code. I have a session running on my machine at work. I'd tap into it I say I want you to clear your context. I want you to pull mean I want you to get to a new working tree and I want you to help me diagnose what this problem is within about four minutes I knew that the disks that were Serving the ECS the EC to our ECS images were full because we had not been pruning them after we would deploy and I had cloud go in because it also has my AWS credentials as part of it and Said go fix it and went ahead and fixed it I didn't even have to restart the system. I had the whole thing back up and running correctly in five minutes Yeah, the classroom basically or whatever whatever yeah, whatever the container service equivalent was but that was it, right? And well the risk was actually pretty low and the likelihood of failure was almost a minus skill because I knew that I could do Everything from the command line on my machine through the app. It was on my phone And so the risk profiles of everything is just different now So when I sit down and I think about what does it take to allow an agent to write code and ship it without Never knowing there's an inherent assumption built in that any failure state is something I can recover from in a relatively short amount of time and Now there's an extraordinary number of assumptions that are involved there and it's not like this is yolo, right? Our build infrastructure our GitHub actions everything that happens post commit is methodical. It's intense going back to what was your initial question We don't do tickets. We do features and oftentimes because of the speed with which we move we if there's not even time to build a UI Our designer is often catching up to engineering We have officially broken the entire organization because engineering lose faster than sales and that is a wild thing to deal with on a business level Like it's when people get fired, right? Because there's too many engineers for the sales team and that's so rarely ever happened because there was always a gap Yeah, I mean the typically the lead time from sales to engineering is six months. Yeah, what do I tell the customer? We need them. We need the feature. Yeah, yeah, and I'm like it's already there Just signed the contract. They were like it wasn't there two days ago. I was like I know we shipped it They're so not used to that. Okay, especially if they're old school and they're used to the giant lags Yeah, well, and you know, so the one thing to say is the mentality of engineers has to be totally different my engineers have to understand the business And I don't let my engineers just focus on tickets as a matter of fact That's an amazing way to fail catastrophically at this point Instead I say here's a thing and I want you to understand it and I want you to do it and the engineers that are working for me at this point Which by the way none of them have professional experience before they started working with me These folks are making really strong judgment calls and that's all I want. I want good judgment, right? The model can and the harness can figure how to write the code So is a large part as a portion of your training or cross-training in the team Helping everyone with judgment is that become an increased thing because since they're going so much faster You there's less time to talk about things so people have to be able to make more decisions on their own Which means they need more agency which yeah, yeah, there's one amazing way to let people build better judgment and you let them fail We ship something this morning and I had actually started work on it and then had to move on to something different And so Mika who's officially he would be classified a junior engineer, right? It will have been out of college for approximately a year in the next few days Mika's making decisions at a senior level but he shipped something this morning and he over looked something pretty major Re major yeah, I halfway knew it was going to happen and I let it ship anyways and That is one of the things that I allow my team to fail I will let the problem hit production is once again it's a risk assessment right because you've done it and I've done it There's nothing quite exo as exhilarating as realizing you just ship the nasty bug, man Everything is being destroyed and it's your fault That is absolutely ingrained memory that you cannot escape and forget it becomes a part of who you are of your Identity whenever you do that and so my leadership style actually relies pretty heavily on that I know that I'm tossing people in every their heads and I do and But I also tell them hey listen, you're probably gonna fuck this up Yeah, and when you do I'll be here. You just come find me. There's no blame I know you're in over your head You know you're in over your head and I happen to know that they're sharks and they have lasers You don't yet. Yeah, and so it's like challenging so yeah You're giving your challenging people to grow and rather than Coddling them through every mistake that they'll never remember because I'm sure if you told him walked by the desk or the equivalent The analogy of walking by the desk and go, oh, what you working on? Oh, yeah, you probably want to check this so that doesn't fail because that would totally broke production They may not even remember that in five months or you know two months So but basically the boss just told him a rule But that's they don't really maybe another deep why I'm gonna ask real quick is the assumption here that you have solid It sounds like it got past testing in the scenario So that means that the testing was lacking for that particular assumption and then wasn't as important by the way that Production can fail back do you need in order to give you the confidence that this isn't going to ruin the company Do you have to have some other structures to be able to roll back automatically or is there a stuff like that in play that you're relying on? So in this instance is actually wasn't a bug It was a missed part of the feature. So this was a little bit unique But honestly, I mean, that's okay. Here's the interesting thing It forced us into the decision of whether or not we wanted to address it and we decided not to address it and yeah You end up have you find yourself in these situations sometimes where because you can't Necessarily cover all your bases you find yourself making decisions you might not have made initially in this instance We were basically changing the ownership of a certain type of object within an agent So previously it belonged to the user and we realized it needed to belong to the agent and so we fixed that right now All of a sudden these things belong to the agent But there were a bunch of them and they previously belong to the user. So we had to make the user's core agent to own them But if you were using a bunch of these things in another agent, they were no longer there and it's oh There's no elegant solution for this. So how do we get things from the PR to production? I think is a great place to really hone in and the first thing you got to know is that whenever our in our Environment whenever you issue a PR stand up a copy of prime for the whole PR It is a somewhat expensive way to run things and it also catches so many things immediately now at the same time That we're standing up that copy of production. This includes the database. This is everything right is a true copy of production We have just a boatload of tests and checks and all kinds of things I think last time I looked there's a 15 or 16 different workflows that get kicked off We've got multiple audits. We've got I mean look we have tests running I don't even know what they are. They just run like clawed was like I don't know we should do this I was like excellent. Let's make that happen now It took me about a week and a half to literally terror our old testing infrastructure out and put this new one in But it's ratcheting we have rough running to make sure that things continue to get more and more You know covered we have unit tests. We have end-to-end tests. We have playwright tests We have a bunch of we have the linters that run even though I think they're useless and so we catch a lot of things Before they even hit before the production or before the that we call them PR environments before the PR environments are built We know there are problems often one of them is we double check that we don't end up with a too many Olympic heads Right, I hate Olympic is literally one of the most awful pieces of technology on the planet if you don't use it Right, it's so we've been forced to use it right Olympic does our database migrations by the way So that's the first line in defense, right now I will take one step back and say that every single engineer can also run the entire stack on their machine That's you have to start there if the engineers have a table stakes You know everything in a way production is a copy of our local environment because even though we don't have the data Like I'm not going to let you put production and buy data on your environment locally you have like all the Docker images that get built on your local also get built with different environment variables in production. And so you have to be able to run everything locally. We also make sure that the model knows, hey, you need to make sure that there's unit test coverage for everything you do. You need to make sure that you're covering, you know, playwright end to end tests. I'm very, very big on, now this is because we're primarily a web app, but I've also done it with mobile now. I want the model to run the actual app locally and double check that everything it built actually works. And so it's not just a set of tests locally. It's also, hey, I want you to fire up a playwright MCP session, and I want you to click around and double check that this actually does what you say it does. Pollucinations are real and the only way to really catch them so that they don't become catastrophic is to literally make the model prove that it's not a hallucination. So that's our testing infrastructure. It's the best I've ever seen. What are these, what are these tests using may files? Are you, you got, are you using testing frameworks that are not fairly necessarily, but trust testing, tooling that is, we use just for TypeScript and then we use, I don't know, one of the Python. Our back, our entire system is just TypeScript and Python. I'm a big GitHub actions fan. I've teach GitHub actions courses. Oh, GitHub actions. Everything's based on GitHub. And GitHub actions locally is often problematic and difficult. And I generally don't use it even though we have, oh, we don't, I don't run actions locally. Yeah, we don't run actions locally. I tend to throw stuff in the agents file, like always run these three linters and then always make sure they're in them, make a make file so that everything you do is through a make file, because I don't know if this is advice or the way that I do it, and I'd love for you to, your opinion on a lot of the things I'm doing, because often if I go fast enough, it's making so much stuff. I built a Mac app over the last month, a recording app, because I'm so particular that the thousand different options for recording screen and video is not good enough for me. So I had to make my own. I agree 100%. And as a part of it, it started to have all sorts like I need to, it needs to do a dev build and it's got a prod building that copies it into my this place. And so I started to say, hey, look, you're just writing a lot of commands for me that I don't even know what you're doing. Could you make sure that all that is always in a make file? And I made that a role in my agents file, and it started doing that. And I said, okay, always run these linters when you're done at time, at every time you edit, always run these linters, always use this make file. And because I started to realize that I couldn't do anything without it, right? I couldn't sit at the command line and know how to build my app because I'm not a swift UI guy. And so I started to the point where I, those are almost table stakes, enforcements for all of my projects where I want the agent locally to behave a certain way like running test. So the challenge is because I'm using GitHub actions, I end up with it just writing whether it's a compose file or a make file or, you know, whatever, I'm always curious what other teams are doing. Since I feel like there's this pull to pull the testing into wherever the agent's living. So if you're running a harness locally, you probably need to run the test looked at because the agent needs to run them. Unless you plan on going and getting a new coffee and you just tell the agent, please run all these and GitHub actions and let me know when they're done. And then you have to wait 20 minutes because we're 30 minutes or longer, depending on your test harnesses and stuff. And so I was just curious what you do. I actually have, it's publicly available on the GitHub. If you want to go see how I started project, this is it. And even I was a skill that you can use. So I call this the Delta V launchpad. It has a cloud skill that'll basically set this up for you. It has a make file. I do everything with make. I'm like you. I got completely just dead tired of okay. I'll do this again. Guys, finally, it's just go use the make file. Make files, you know, the Docker configs are all composed files. It's all pretty standard stuff. This is all none of this is rocket science. We all know how this is built. Yeah. That's why I just picked something. Yeah. I think a lot of the people in the my guild, the genetic DevOps guild, they're, you know, there's a wide range. It's in there. There's people that are like very new to this. They're dipping their toes in the co-pilot or cursor, maybe cloud code. And there's people all the way on the other end that are, you know, they're using they're using plug-in marketplaces to manage their skill deployments to their teams and centralizing the skill management. As while at the same time, they're adding more and more agents to their post commit workflows to self-improve things on occasion, which is a topic I want to get into at some point about the idea that once you sort of matured yourself to, you know, you've got your agents reviewing PRs, you got sort of like the the table stakes of agents post commit where you've got it, you've got agents looking at your PRs, you've got agents, maybe you're lucky enough and you've got one can help you troubleshoot or something. Yeah, but here's one of the questions. Everybody should really ask, do we have a way to do this in code? Because if we do, we should do it in code. Yeah, it's funny. I talked to you because sometimes they're like, I want to build an app. I'm like, all right, tell me about it. And this one guy told me about, he runs his pull business on an Excel spreadsheet. He was like, do you think I can turn this into an app? I was like, absolutely. He goes, where should I add AI? I was like, nowhere. It runs on an Excel spreadsheet. It's deterministic. Do you have anywhere in this process where you want a non-deterministic choice to be made? Because the answer is no just run the code. But yeah, I mean, look, I tend to drive certain types of engineers insane because I kind of don't care. You know, you're like, well, what are you using for your testing? I'm like, I don't know. We're using something. I know there's tests there, right? Who's looked at your test to make sure that they're valid? Nobody. How do you know your tests are good? Well, I have about 80 other things that check, right? And so I don't dictate to my team the tools, right? I do say, you're going to use Docker, right? You're going to use Docker Compose. You're going to use this make file. But that's more just for efficiency sake. You know, one of my engineers, he's way smarter than I am. He uses codex. I think it's a terrible idea, but he uses codex and we fight about whether or not chat GP tier or cloud or better on a daily basis. And I tell him he's still wrong, but whatever, though it is funny. We get to watch the models leap for each other. I really don't care what the tools are as long as they get the job done. And that means I also change the tools often. Well, no, but I mean, it brings up a good point that sometimes the reason I ask these questions is because in the question of, you know, think of some of the things you're using in CI and just rattle off some tools because we can be vague and hand wavy about a lot of these concepts. But I always like to get into the nerd details. But the reason I asked that is because often the answer is, yeah, Docker Compose files and make files. In other words, there's nothing new in there. There it is exactly what it always was. I personally, I don't know if this is the general advice you would give, but I personally advise a team to guide their AI to make the choices on the toolings that they know rather than just let it make because in theory, if you asked that AI before you've greenfield something, what language it thinks it would use. It might, it probably will prefer Python or TypeScript, but it might make a bold choice. And that doesn't mean it's right for your team. And so people I think are sometimes maybe not so much anymore, but they think they were looking at this. Is this an opportunity for us to reinvent the wheel on a lot of our tooling? And my argument to me isn't, no, because at any point, look, the model uptimes aren't 100%. So if at any point, you're stuck and you have to read something, it wouldn't it be nice if it was in a thing you knew instead of some completely foreign like, I'm not going to have my AI building a pod man interface to my containers when I'm the Docker guy. So if it starts to do that, I'm going to guide it back to the places, even though I may never, ever look at that Docker file after the first creation of the PR. And I don't know if you have standards like that, but I've always been imagining that teams need to actually start even polyglot teams. They need to kind of settle in and say, look, these are the languages we would really prefer a you stick with an AI and not just don't be co-balling our production. I've got this Instagram channel that seems to be blowing up and I piss engineers off all the time because I say things like you shouldn't be reading code and you shouldn't be writing code and there's quite a few engineers that are very, very little. The other one I had recently is like, yeah, just make sure you're testing around kind of no matter what they are. And, but I had a few people get in like really get nasty arguing with me. Yeah, finally, identity, bro. No, no, they were right. They were right because my assumption was different. So I've had a couple of people come in and say, this is insane. How can you do this? It's too risky, and I've had a lot of people come in and say, well, then it turns out that they're building embedded systems. They're in health care. They're in banking. They're in some kind of high security field. I'm like, yeah, don't listen to me. Right. But if you're building web apps, you're building mobile that is just an extension of the web. Yeah. It's already been done. Just use the standard. You don't have to reinvent that wheel. You have a very valid point about your assumptions because both of us have been operating. I mean, I've been in cloud native for over a decade. The things that we learned about distributed computing, microservice architecture, API contracts, all those sort of the things that kind of evolved over the last 15 years. I've been living and breathing that at the near tip of the spear. So when I meet a team that they have 3% test coverage, they're not using a type say for memory-safe language. There may be yellowing some old PHP stuff, and they've got a very monolithic approach. And then they're like, how we can't do this? We're having all kinds of production failures. And I look at them and I would evaluate and say, yeah, that's because I love this phrase lately is that AI just amplifies the team culture. And your team culture is of 2010. You need to modernize a lot of your culture before you go fast. You can't go fast without the safety and the safety was never there to begin with. And so I do run into that on occasion with teams and it's surprisingly a number of teams, maybe not embedded in healthcare, but like a surprising number of teams that have very large, important apps that I get to consult with. And I cannot believe the, I would call it the level of immaturity that the code base and the team has. And because we don't have accreditation in our business, right? We don't have plumbers certifications. So like I can't look at them and assume that they know all these things and that they're doing all these behaviors that I've been doing for so long. I have been able to look under the hood at multiple Silicon Valley giants. It's all shit all the way. have. Every single engineering day was, we have the worst stack ever. All of our goods graph. And it's, now your code's fine. There's really only one measure that matters when it comes to talking about your product. Does it work? A follow up question is, does it work all the time? If the answer to those two things is yes, it doesn't matter. Now, there's a third implication now recently, especially with a bunch of the mythos stuff, which is it secure. And that's always been important, and it's always been relatively easy to exploit sloppy stacks, right? By the way, sloppy stack is the name of a future podcast. Oh my God. Yes. That's that's really. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Break down your sloppy stack. No, we're going to, it doesn't. It also sounds really gross. Whether or not your architecture sucks, whether or not your stack sucks, whether or not anything sucks, there's only one measure that you have to apply to it, which is doesn't work, right? And now that decompose is fast and tiny little things. But if it works and it's built on ancient PHP, maybe there's not a reason to fix it, right? I have another follow on question. So we're seeing a lot of the news around the general degradation of code quality as people on board AI. And I understand how it's happening, and I don't understand why it's a mystery on how to fix it. Because it feels at least in my experience, if I have all these review agents and PRs and I'm using I've got a series of skills that are auditing security audits, concurrency audits, performance audits, you know, these are things that used to be very hard for engineers to do because it would quite a lot of manpower. And we would often skip them, especially when budgets were tight and project deadlines were close and continual documentation updates. I'm just thinking going down the list of the things that I can now have agents burn a bunch of tokens on. To me, it doesn't make sense that the code quality would go down. It feels like to me that with enough gates that I'm triggering AI's in that the code quality would in terms of maybe not my impression of how the good quality is, but at least the way it performs and behaves in production feels like it should at least stay the same if not go up. Is that what you're seeing? Do you see that or do you see the opposite? Am I wrong? Or I don't read the code. Why do I care? Well, but you see the outcome of the code. So obviously like the outcome of the code. So if your production, if you were five nines before and you're four nines now, you would notice that assuming you're measuring it. I mean, listen, we're not five nines. We're close. We're much closer now than we were two months ago. You know, it used to be I had to take about 10 to 15 minutes of downtime per any release because of the way our stack is actually no. And because of the way that we release, but I've since fixed that we don't have downtime. As a matter of fact, I mean, we probably are close to something like five nines. But the question is like, well, let me, let me turn this back on to you. Why does code quality matter? And what is it? Well, why? That's a great question. That's a question. I would say what I'm specifically talking about is failure rates and production as adora metric. This is the problem. This is my assumption is when I think of code quality, it isn't whether you tabbed your space. It isn't whether you followed the black linter on your, you know, whatever spec. It is about when you got to production in an enterprise, there will be a security team that does a production analysis of what's going on and they'll find more flags than previously, or you'll start to have more outages or customer complaints than before. And so these metrics, according to the, the news, the media, you know, these things are going up for teams that are going too fast in AI. And that's that some are deriving that as the conclusion means humans still have to read the code. I don't necessarily take that as an outcome of that assumption. I take it as no, you need to put more, you need to put more rigor into the AI process, not back up to the world where we were all reading every line of code. So I was curious on your take. It would be interesting to go back to when the Agile Manifesto was written. Everybody had done waterfall to that point. And all of a sudden, somebody comes out and they say, this is stupid. We should change as fast as we can, right? We shouldn't plan for a whole freaking year. That's insanity. What we should do instead is we should take stock of where we are and take stock of where we want to be in two weeks and get there. I am willing to bet that the performance of Agile at the very beginning was much worse than waterfall. Nobody had done it. Right. Everybody was trying to figure it out. This is worse, right? This is that steroids. I look at the things we are doing now and on my team, that we weren't doing two months ago and I sweat, right? But you know what? My release rate, the amount of code that has landed is hockey sticking and I've been forced into what I'm doing now. I would be willing to bet that we are shipping more bugs than I wish we were shipping. I don't think we're shipping the same bug twice. At least not one that like, have we had regressions? Absolutely. Have we had regressions in this change over period from the old way we were doing it until the new way where we'd done it? Yes, 100%. I don't know that we've shipped a regression since we started on whatever this new thing is that we've been doing. Okay. And then you've only been doing it for three weeks. Is there like a hot tip on exactly that do you add these to a skill? Do you add new tests for every failure? Oh, hey, thanks for listening. And I got a couple of quick announcements about stuff that I've created. A lot of this is free stuff you can get right now. I just in case you've not seen some of my other videos or stuff I've been doing on my newsletter, I've created a GitHub Security scanner that is an open source project. It's called GASA GitHub Actions Security Assessment. And it's expanded to even have more rules. I think I'm about it. Now it doesn't rules on it. It scans your repos, including the entire organization. And the focus of it isn't to replace other linters like the GitHub Actions Linter or Zizmore or other scanners like Poutine that are all really great. It's meant to supplement those and solve my own problems as well as my clients problems. And I just thought I'd open source it to give it to everyone. And I've spent a lot of time trying to make it a solid tool. I actually just spent some today, time today in Fable Model building out some better testing frameworks to test it against real organization repos as a part of its PR testing. But the whole purpose of this is to scan its personal repos or even repos you might want to use or mostly your organizational repos all at one time to give you this report on your stance of GitHub Actions Security. And it looks not just at the workflows, but also the repos settings on the backend. And this is based on my half decade of consulting around GitHub Actions. And there's no agenda there. It's completely free. I've actually been using it to implement and upgrade the security of my own repos to make sure that I'm not as vulnerable to some of the recent supply chain attacks. So the whole reason of doing it was people were asking me to help them lock down their supply chain against the attacks we've seen over the last year on GitHub Actions, particularly around open source repos or even using other people's GitHub Actions in a more secure way. It will guide you through how you can change and improve the security. It won't actually change all the settings for you because I'm not quite comfortable with it going in blast radius of your entire org. But go check that out. It's on my GitHub and the link is in the show notes. Next, I had in July a GitHub Actions Security workshop for free because chain guard sponsor me. I was actually going to charge for this workshop because it was hours worth of hands-on learning around how to use your own agent or your own hands to actually lock down your GitHub to make sure that you're not vulnerable. So it's walking through what this GASA tool does, but also why these settings need to be locked down, why you need to care about those particular settings and what happens when you maybe do one thing in a workflow and another thing in a security setting in your admin settings and those two things conflict and cause a security potential risk that GitHub isn't probably warning you about. So it's meant to supplement all the current stuff out there that really just focuses on the workflows, but don't actually also consider the admin settings of your workflows, which are behind the scenes. People can't see those settings, but they're not usually easy to find. So anyway, you can go look at that workshop also free on my website. Thanks to chain guard for doing that and making it free for everyone. And finally, if you didn't know about my agentic DevOps Guild, we have been meeting weekly since March. We I think we're like 25 weeks of weekly meetings and we're now over 50 teams of people in there solving problems for their team on how we're going to manage agents, how we're going to manage the security and the sandboxing of our AI for platform engineering use DevOps, anything related to cloud and Kubernetes. If you're somewhere in an ops role, this was the program designed for you. It's not just meetups, but I'm also creating courses. I've got two courses in the works right now where videos drop on a sort of a random basis, but based on what comes out of our meetings, I turn that into lessons for everyone to adopt and also that comes with workshops that we do, including this gas of workshop that was last month. And those workshops are meant for the guild. The guild will always get those for free, even if someone doesn't sponsor the workshop. But go check that out also on my website. And I hope that helps solve some of your AI adoption problems back to the episode. We have to be absolutely obsessive about tests. Yeah. I hated tests. I hate tests. I hate tests as an engineer. People that are like test-driven development, I would always be like, you're at ease. That is the stupidest thing. I cannot think of a better way to grind an engineering team to a halt. We've got to ship this fast. I'm going to save way with spectrum and development that I am with test-driven. I'm a shiny person and none of those things give me the dopamine hit of shiny, which is why no one pays me professionally to develop full time. But you know what? I get test-driven development for free now. Yeah. I don't have to worry about the - Yeah, let me throw this at you. I was working on 15 different bugs at the same time the other day, okay? I was doing so much that I locked my machine up by filling up the disk because apparently, if you run node, what is it? The MPM install and the tight, the Python equivalent, that can be a lot of space. And I only have a half a terabyte on my work machine 'cause I decided that's the fine thing. That code is small, and that you do. And I was building Docker averages, yeah. And the next thing I know, my machine just stops. Not slept down, it just stops, right? But in the process, I had also opened up 15 PRs. I locked up the entire stack. I accidentally flooded the GitHub actions, overloaded the lambda that runs our database cloning where we clone production into the PR environment. I found a bug in the process, which was very nice to find. But it was not the first time that it happened where I basically launched this clone storm that locked our entire PR infrastructure up. Everything was still running, but you couldn't do a new thing. I've never tried to land 15 PRs at the same time before. Have you ever done that? - I didn't close. - Has that, have you ever heard of anybody doing that? - Yeah, I mean, there's the whole new, what's the PR gating thing where you combine them? - Yeah, it's like merge streams or something? - Merge, yeah, merge stacks or-- - Yeah, it's basically what GitHub just told me to do. - Oh man, get a sloppy stack this hard. But listen, yes, of course it's, yeah, maybe it is degrading. But nobody's ever done this before, man. But we were doing it before Anthropic and before OpenAI. We were doing this style of engineering back in July of last year. Anthropic and OpenAI did not adopt it. Nobody's done this, man. And so, yeah, are we finding things that are degrading a little bit, sure. - Yeah, how long it takes me at this point to put a fix into production, if I notice it, 20 minutes, 30 minutes? - Yeah, assuming I have to build the images? - I recorded a very similar episode to this topic yesterday with someone from GitHub Dexed and then someone from Microsoft Research. And they're both working on the agentic workflows feature set for GitHub, which is currently, I believe, in beta. And I'm very excited about it. I thought it was very weird at first, but after spending an hour and a half with them, I am all on board and I'm very excited about spending more time with it. He talked about that he's also working across 22 projects with multiple PRs in each project and that he does everything, because they're GitHub, they do everything through co-pilot, everything in GitHub. And so, he spends most of his day as a researcher that is a developer researcher, as I should say, prompting through GitHub.com/whateverTheAlogenticPages where you can prompt like you're in Code GUI, or Cloud Code GUI. And he was saying, you know, he spends a lot of time on his mobile doing it. And sort of talked me through his day to day, and I was like, man, I feel so antiquated, 'cause I'm still at this habit of, if I think of something, I go and write it down on my phone, but I think I'll do that when I get back to my computer because I'm just so not driven to use dispatch, or I'm just not there yet, yeah. I know I have it, and I just don't default to it yet. It is, oh, I can kick this off right now. I don't need to wait till I back at my computer. - So this is what I do. I have two machines that I write code on. One is this laptop here, and one is my machine back at work. I typically use my machine back at work for this. I have remote sessions, or remote environments, or something that turn automatically in Cloud Code, and then I also have auto mode turned on back to fault, right? - Safety off. - No, at least safety on. Auto mode is not-- - Oh right, sorry, auto safety mode, right? I'm sorry, I'm thinking of auto mode. - It's legitimately called, there's planning, except that it's whatever the default, normal default state is, and now they added auto mode. Auto mode actually has a safety classifier in it, and so there's effectively another agent that's over here running that says, is this dangerous? Is this dangerous? Is this dangerous? - Yeah, no longer can you do that. - Right, right. - We'll just smack it, yeah. - Sorry, you don't, real quick. You don't enable sandboxing in the-- - Actually, auto mode may be some kind of sandboxing in the background. - You could call it that, but yeah, 'cause it's gating, there's-- - Gating tool access, basically, right? Like it's a programmatic deterministic gate, at least as maybe not auto, maybe part of it's a little bit of a mini LLM or something in there. - Not fairly deterministic way to do it. - Yeah, but then there's also, then there's your unsafe mode. - I'm hands on. - Okay. - So the danger with unsafe mode is actually auto mode and unsafe mode are very similar, but there are certain things that auto mode will not allow the agent to do, and one of them is just go find a module to use, and that is super duper important right now. The number of tool chain attacks that are currently in flight are insane, and the security landscape is, it's actually one of the things that keeps me up, thinking about it, did we ship something that has a security vulnerability that was intentionally put there, so that our stack would be exploitable later, - That's sleeping on my machine. - And is it sleeping on my machine? - I'm waiting to take my stuff. - Yeah, but I have these remote sessions running, and I'll be sitting, I used to not sit on the couch and watch TV with my wife because I felt like, I needed to go be doing something, and I'd have these ideas, I don't love watching TV, but I'm like being with my wife, and I was like, okay, well, let me go sit with her, but then I had these ideas, and I'd be like, I need to go do this. Well, now I just pull my phone out, and I tap a few things on, into God, and I just put my phone away, and it goes and it tests everything, and does the idea, and then it comes back, and it's, okay, yeah, it's ready to ship. Are you ready? I mean, it's a wild world. - You can't start a new project from the phone, can you? On cloud codes specifically. - Is she in? - Okay, but when you say a new project, what do you mean? For example, if I don't have remote enabled on an existing repo that I've cloned, or I want to start a new code project, and I don't have a repo yet, or a folder, I can't get it to initiate a local new session on my Mac remotely, right? Like you said, you had remote on always, or you have it, I didn't even know you could do this, is actually a pro tip, is just turn that thing on for everyone, all of them, 'cause I always get to slash enable remote in my sessions. - All right, so first off, you have to give cloud code the access to your GitHub repo. If it doesn't have a GitHub repo, you can't use it. Now, if you want to start a new project, you go create the new repo, and you get how to happen, then you come back to cloud code, and you give it permission on your phone to just use that, and then it can. Then you can start the, you can start it in the, there's like a cloud version, then it runs, and you get all the changes. - And you're using the connections feature, that actually connects. - Yeah, I don't remember where it was, but. - Yeah, it's an automatic connection to GitHub from the cloud code web UI. - Yes, that's correct. And now, because of the way our stack works, because I can issue a PR and stand up the copy of production, I actually don't have to have something running on my local, but I can continue it. And so, if there's something going on and I'm like, "Huh, I would, I need to go walk away for two hours, but I want to make sure this thing progresses." I'll just get a notification, I can look at it, want to mable and say, "Okay, go do this, that and the other." So that is a pro tip. You can, if you want, be running through first landing part with 25 pounds of computer equipment on you, and pull your phone out and advance your project again. - I don't see a lot of those weight, exercise weight vests with laptop sleeves in them, but maybe that's the nerd equivalent. - So, if there was, we've talked a lot of high level stuff, and I really, I think this is gonna be really good for people to sort of hear a team that's, I wanna make a point here that I feel like one of your secrets is that you're in tune with the business, which you kind of started with that. Your team is aware of the business goals, sales objectives, like in terms of what the product needs to be, where we're headed, why does this thing exist, what do our customers look like? Because that allows them more agency and allows them to, also, there's a metric of judging risk there. Because like you said, if you were a healthcare, if you were a thin healthcare company, that was, that you may not have this same posture, because you're appropriately dialing up the risk tolerance to the level of the organization that you're in, and that is different for every organization. So I think it feels like it bears worth repeating that if you're not, do not try to shove this round peg in a square hole of, you're in a high safety, you're on a DARPA contract. You probably, yeah, you probably should be doing old school co-pilot code completion with the AI at best, where every line is sweated over, and you're just saving keystrokes. - I've always been the engineer that was connected to the entire organization. It's just my nature. I like talking to people, and I'm curious about how the business works. However, that is a skill that engineers are gonna have to have in the future. At least to some degree, when we started this, the very first thing we realized was that our infrastructure was the first gate. So we went ahead and we fixed our infrastructure, and we thought, great, now we can move forward. But then we broke product. Product could not keep up with us. There were only three engineers at this point. - Product could not keep up. - Then we broke design. Design is consistently behind in our system, and it's not design's fault. It's that there's now four other engineers, plus our designer who actually commits code, right? Because that's the only way for him to keep up with us. Then we broke sales, right? I would have to get on sales calls, and consistently during the sales call would have one of our sales guys be like, yeah, we're gonna have that in a couple of months, and I'll say, no, it's in the product right now, right? I have closed deals, because it was already in the product, and I was the only one that knew. - Yeah. - Right? - It was fresh. - Yeah, we don't have much marketing, so I don't know if we've broken marketing, I suspect we have. But when you're in a position where the engineering organization is now the fastest organization in the company, the only. way for you to move safely as a business is for your engineers to know how the business works. Right? So before we ship things, the engineers at work with me know, hey, do we have any demos coming up? Because that's a real question. We're small. We have to ask that question, right? Are there any important things happening today? Because that's something else we need to know. And so these are the questions that engineers typically don't ask and they're going to have to. Engineering is different now because the speed at which engineering can progress is effectively putting pressure and force on the rest of the business and breaking it in the process. And I've never seen anything like this. And that's how I know it works. Yeah. You're describing this as what I would maybe label as a pro engineering culture where a lot of the messaging we see in the media is around cost optimization of reducing engineering. And what I'm hearing that the language out of you, which is clearly part of the business culture of your company, isn't looking to save the money. But because you're so efficient, let's reduce you down to two engineers instead of five. Right? Let's give it a designer and burden someone else with that workflow. I don't hear that out of you. And that's sound I'm assuming that's pretty deliberate. It's a deliberate strategy to not starve the engineering organization. Well, I mean, to be clear, the idea that what we've built is what we've built with just four or five engineers is observed. There's no way I could have presided over this thing being built with the number of engineers that we have with the without a yes is that AI cost savings only in so far as it is an accelerator to your individual team. AI is not going to be cheap. We keep on that out. We discovered that recently. Yeah. The minute Claude Maxx and the $200 equivalent for open AI goes away. Small companies like mine are going to be in a world of hurt. It's too expensive. But the options that we can afford to do it any differently either. So yeah. That leads to a question around cost. Do you? I have this diagram which want to show you real quick. I'm trying to explain in my courses to people that are, you know, not near as advanced as you all are yet and or as experienced. And I'm taking the project management triangle, which was traditionally it was the rule was there's three things, leverage you can pull, you get to control to you, meaning management and you want to control all three. And that is quality, cost and time. And so the whole adage was you might be able to, you know, you might be able to reduce the cost and and resources. But that's going to cause the timeline to stretch. So you move to parts of the triangle. The third one's going to be affected. And I'm starting to correlate this to what I'm maybe calling like the AI engineer project management triangle, which is a sort of an internal metric that we assuming there's not unlimited budget, which some teams are very lucky and they can just token max to all day long that we really kind of get three new choices we didn't have before that is model choice harness choice and where or when to use AI because obviously you could have AI robot. You could put an agent into production and have it Ralph looping every second of the day, constantly looking at stats and production to look for any potential failures and it will burn thousands of dollars of tokens a day if you let it, right? So not everyone gets to choose max AI in every part of their software life cycle. So if we get to choose model, which effectively not only is of time thing because kai koo is way faster than opus, but it's also very much a cost thing. And then which harness choice, which is maybe more around functionality and ease of use because cloud, I feel like it's so so far ahead of everyone else that it leads to a lot of new functionality that we don't yet have on some of the other ones. And then when and where to use AI, which some organizations maybe don't even let you put it in a production yet for helping with troubleshooting. And then there's these esoteric things of the environmental impact, the privacy impact, and then the political impact of certain things. I have members of my community that are in Turkey, for example, where they're relegated to not being able to access all the models or they have privacy rules where they're not allowed to move this data outside their data center. So they have to then be able to somehow house local models that they have to run open weighted or whatever. I was curious if there's anything in here of value that you would think that my idea has merit in the fact that this is a new paradigm that we've never in the past. I don't recall the last time I had to worry about cost in my daily activities that might blow my budget as an engineering manager. This is a new thing, like you're allowed, but that wasn't happening in a day, right? So first off, the most important part of this is the harness. There was a Stanford paper and the Stanford paper came out and basically said, you can improve the performance of the model by 6x if the harness is correct. And so harness is really what you need to focus on. And while cloud code is one of the best harnesses in the world, if not the best, they're still an extraordinary matter room for optimization. So our models are harness. So we actually, yes, I code in cloud code, but in our production environment, we deploy agents. This is what we sell is this agent infrastructure. And our agents write code. Now our agents do not get tool calls. I don't let the LLM touch many tool calls at all. What I do instead is I package, we package the tool calls into an agent SDK. And the only way for the model to touch the tool is through writing Python code. Now, what is that in doing for us? Well, number one, most models are really, really good at writing Python code. You can actually deploy small models to write Python code, right? But we realize we have this immense cost saving right now that we have not leveraged and we're going to because if our agents are writing code to express themselves constantly, why know how to save code? As long as I can find that code again, I can actually reuse it. I don't have to spend the tokens recreating it, right? And all of this, I've built this Instagram agent that I've been using for my Instagram account. And I actually built my own Instagram integration in cloud code and I moved it into my agent. And I said, whenever you do anything with Instagram, go use this code. And so what I found is that it has been using it and improving it daily. And I realized, oh, I think this agent is now using less tokens because it just goes and runs the code it's already built. And so that's one of those things where it's, oh, well, if the harness can be used to make the model better, the harness can also be used to reduce the number of tokens that are necessary to run on a daily basis. And so putting the harness at the center of that thing is actually the way to look at it, right? The central connector between the cost of the LLM, the time that the LLM takes to run and whether or not it's safe to run is the harness. And in point of fact, you can even structure your harness such that you can observe the inputs and outputs to the harness without ever leaking secrets to the LLM. Because one of the other things we do is we proxy everything in and out of the harness that's running on our remote system. And since we're proxying it, I can actually inspect the traffic. And I can do that with code. I don't have to do that with an LLM. And it's fast. I know how to do it. I know the things I'm looking for. And so now if I start seeing the harness producing stuff that it should not produce, then I can smack it down. And conversely, I don't have to give the harness all the secrets. I can inspect the traffic on layout and insert the secrets and flight without ever exposing them to an LLM. So the real, the center of that triangle is the harness. It is the most important thing in what we're doing. Now, are the models amazing? Yes. But now the models are good enough to focus on the harness. Yeah. Is that using light L as your proxy? No, do you know? No. We rolled our own. Okay. So in terms of bringing in the LLM, we just bring the LLM in through the, I don't like light LLM. There were at least some significant issues with it previously. We use the length, length, length and graph library. And we just make sure that our LLM is conformed to that particular structure. We've got teams that I'm working with that are trying to figure out when it comes to local human agent harnesses that, you know, they're trying to understand usage per person and trying to figure out, give model access as well as maybe inject things at some of the sandboxing tools are also learning how to inject things on the fly in the H2PS. So the last question, I was going to see if I could wrap this up with the last question. And I'm going to show you a graph again to them another one. I try to spend some of my time, I'm trying to, and I'm really bad at it. So I'm forcing myself to do it. So what I have on screen is something that I'm working on calling the AI engineer maturity model. But it comes from a difference aspect of then every single other one that I see on the internet. And I spend a lot of time looking at what everyone else is talking about of like, how do I onboard my team to the skill, the human skills they need to really complete the life, the software life cycle of where all the AI is belong, right? It's not just about cloud code on my desktop. There's lots of other opportunities. And of course, the industry is now starting to figure out that if you speed up the number of PRs you're shipping, but you're not improving the things after that, then you're just going to have a giant bottleneck in your PR stack and production won't actually get faster. But the screen that I'm looking at is a fictional screen that on it is the idea of over time as you implement different human workflows or different human skill sets that the team ramps up on. We assume that your overall team productivity improves and there will be times, like you said, when you first are doing certain activities that team productivity actually declines, like the day that you implement cloud code for the whole team and they're all using it, that's a rough week, right? That's a really unique thing. You need to give people time to onboard. So there's probably going to be a lower productivity. And then there's other points where I see teams that maybe they've they fully adopt agent harnesses locally, but they really haven't done much in CI yet. They're barely even doing PR review agents, they've maybe not even using that yet, and they realize now that they have the CI backlog. So I would argue that becomes a team productivity dip because now they're spending all their time staring at code the AI built and they haven't really kind of leapfrogged to the next level. So I was looking at this like imaginary list of things you can do in a team to improve the overall productivity of your software deployments. Do you see like activities or certain things that are sort of rise up as the obvious things besides just using Cloud Code in your desktop? Is there activities that you think are a big tell on whether the team is mature or not in their AI software life cycle? - I hadn't thought much about it. I've been partially because I haven't encountered teams that are doing it the way we're doing it. - Yeah. - I promise you I've had a lot of people look at me directly and tell me I'm insane. They're like, this isn't real software engineering and then they come back. They're like, oh, here's what I can tell you. The end of that graph looked like this gentle line that just went up and there's a point in there where that gentle line just becomes vertical and that's the point where you do have agents fixing things without human input. We moved into, hey, it was last time many of y'all actually looked at the code and all my team was like, everybody didn't want to get caught. I was like, oh, 'cause I haven't looked at it in months. - Yeah, they were like, oh god, okay, thank you, thank you. They didn't know where you were going with it, right? Like you were, you were, you were putting your question in posture syndrome that jumps out when the juniorist member of the team posts something in the engineering slack that calls out something the most advanced engineer's done and you're like, oh god, I'm not even a real engineer. And then you're like, oh, wait, you had caught look at that didn't you and they're like, actually no, it was codex. But is everybody using AI full-time? The next step is, what was the last time anybody read code? Not just write it, read it. Do you have full-CICD? And then beyond that, it's as simple as, you know, I'm sure that you could throw some milestones in there, you know, what's your how fast is it to fix a bug, right? Because in lieu of a perfect release, how fast can you fix the release if it goes sideways? By the way, you did ask one question, do you have a single button rollback? We do, but it's only good for 10 minutes. So we got to be real careful. But then finally is, can the agents just do your job for you and under what constraint? And that is when I think you're at full AI maturity, that when it is safe enough to let agents move in your system without getting human interaction, you are now scaling yourself infinitely because every step change that you add to those autonomous agents is an infinite scalar for the rest of the team. One thing I will point out then, we can close on this is when you make mistakes in this style of engineering, they're big. I'm not talking about shipping bugs, okay? I'm talking about a misstep in the architecture. I'm talking about a misstep in holy shit. How did we manage to ship that migration order on the database? When you make a mistake like that in this style of engineering, it's not a self wound, it is a decapitation. And so every engineering team will hit the point where they're like, oh my God, what have we done? And we've done it twice. And both times we ended up having to regress the system by two weeks, which doesn't sound like a lock. And so you think about the fact that we have commits that have been 60 commits behind. We've had PRs, 60 commits behind because they were five days old. So the cost of the catastrophic failures is higher. And that's the one that you do have to think through. Do you think about that in terms of either different tiers of risk in the way you approach AI and different repos or different parts of a repo where you're like, well, this one's dealing with the database migrations. So we have extra safety involved with that. Yeah, so we do have, we use an owner file. We need to be using an owner file. I just keep forgiving to put it in. But there are very specific-- Confession is there if we say again. Get confession hour, this is confession hour. Oh, believe me, we're not perfect. Everything I've told you has been relatively true. And maybe I've waved my hands once or twice, right? But yeah, we need to put the owner file in. If our system prompt changes, I need to know about it. If our list of tools changes, I need to know about it. If our-- but we ship multiple database migrations in almost every single PR. I mean, that cannot explain how many times we have metamorphized the database in just a few days. But there are files that if they change, I want to know about it. By the way, one of those is going to be, like, what are your imports? But node modules are you bringing in? What Python libraries are you bringing in? Because those are important as well. Yeah. I can see how that would trigger different-- additional safety checks and GitHub actions, where-- I mean, I do at times make directory scoped actions that are-- it's also really good just for any action. This is an approach of a GitHub actions. A lot of times, your actions that run all the time don't need to run on-- they only need to run certain types of changes. And GitHub actions turns out it can actually have a lot of filters involved so that you can say, hey, don't run the action-linter unless the action files changed. And that's only in this one directory. So there's no even reason to even fire it off unless it hits data. I probably could-- if I probably paid you for an afternoon, I could get so many more efficiencies out of our system, dude. It's like, I'm not-- I'm not infrastructure. So-- Yeah, at this point, I'm just a skill. I'm just a skill. I just have an actions workflow skill. My new rule is every new conversation I start with an AI needs a skill. It's just a one-off question fine. But if I'm planning on this going to take an hour or two, we're going to have a conversation about it. I'm looking for a skill to give that agent enhance its ability because I personally have noticed on my project in the last two months that the skill in every case has always improved the outcomes, or at least got to be faster to the outcome, whether making sure the skill is teaching it the latest version ago, making sure that it's adhering to the latest skill patterns, or whether it's the front-end design skill that everybody is using, or whether it's a GitHub action skill that enforces certain linters and certain security patterns that I behave with. I just feel like there's always a good reason to have at least one skill in your conversations. Protit. Hey, man. So what's the one thing you're going to excited to do next? What have you not done that's on your to-do list of I want to try this out? Turn the agents loose. Get them to fix things end-to-end without me ever knowing. Continue with some improvement. To be able to have an agent pick up a report from Slack, figure out if it's real, create the issue, fix the issue, commit the code, build the PR, test everything, merge it to main and ship it. And I never know where that starts happening. Our engineering team of five will look like an engineering team of 80. Check out "Agentic Workflows" in GitHub, if you haven't seen that yet. That's the actual term, "Agentic Workflows." Because it writes, it's like a full safety version of putting in a prompt in a workflow without all the risky behaviors that everybody's using right now. There's like sandboxing. There's all these different extractions. And one of the things that you can do is you can put it on a kind of like a Ralph loop. According to what I heard yesterday was they designed their system so that it's not all Monday at 9am. And they distribute these things out. It has inherent rules inside it. So if you ask it to be an issue reviewer, it won't do all the issues right now. Like it's not gonna overload you with all of this work, which is what a lot of people I think do. And initially, the AI will, some of a bunch of work. - I will say this. If Copilot is my only choice, I won't use it. - Yeah. Well, then it's got to do with Copilot. You can use Copilot. - Okay, good. - So, yeah, you pick your own agent. - Okay. - Their framework is agent agnostic. - Actful. - Yeah, I may try it. - Yeah, I think the push button features in GitHub are all Copilot, right? You turn on the PR review and your settings. These are just features that happen to use Copilot. This is more of a hobby kit approach, where you're at the command line with this stuff and using YAML files. I'll show it to you someday if you don't ever get a chance. Mike, thanks so much for being here. Can you give us your full name and title and company? - Mike Rollins, CTO of Relifying and Family Digital. And also find me at Rollins.io online. Instagram is Rollins.io. And I think that's the major point. - You talk about AI on Instagram. So if people want to hear more thoughts from you, I would recommend Instagram. - Awesome, man. Well, thank you so much for being here. And I'm gonna definitely have you back on 'cause we're gonna talk about you pushing the envelope even further. - Let's do it. - Thanks for joining us. And I'll see you in the next episode. (upbeat music)