Go back

Why Vibe Coding Might Get You Hacked! Why AI Tools Act Like Junior Devs | Eric Müller

48m 44s

Why Vibe Coding Might Get You Hacked! Why AI Tools Act Like Junior Devs | Eric Müller

In this podcast interview with Eric Miller, an engineering leader and security expert, the discussion centers on how AI is transforming software development. AI tools such as Copilot and Claude offer significant benefits by accelerating tasks like writing boilerplate code and generating unit tests, effectively acting as a "junior developer" for experienced engineers. However, risks arise with "vibe coding," where inexperienced users rely too heavily on AI, potentially producing low-quality, insecure software. Key concerns include AI hallucinations, security vulnerabilities like "slopjacket" attacks, and over-trust in generated code without proper vetting. Miller emphasizes that AI excels as a prototyping tool but cannot solve novel problems absent from its training data. He advises treating AI-generated code with skepticism, subjecting it to static analysis, human review, and strict permission controls to avoid security breaches. Ultimately, the responsibility for building safe AI systems rests with producers, not consumers, and healthy engineering teams should leverage AI to enhance—not replace—human expertise, focusing on higher-level design and architecture.

Transcription

8185 Words, 44139 Characters

English
We'll be back to an hour of innovation. I'm your host with Lotion. Today, I sit down with Eric Miller, engineering leader and security expert that work and co. We talk about how AI is reshaping the way software is built, the promise of AI assistant development, and the dark side of hallucinations, security vulnerabilities, and over trust. Eric shares how to manage healthy engineering teams in the age of AI, and what makes a great leader. Stick around until the end for Eric's honest stake on the AI future in security, and why it's not about replacing humans, but building the right car trails around them. As always, if something resumes, drop us a comment for sure this episode. Now, enjoy the conversation. Welcome, Eric, to the podcast. Thank you for joining us today. Yeah, thank you for having me. It's a pleasure. Yeah, great. So why don't we start with a short background and story, how you end up in your current role and how you got there, basically? Yeah, so I got going back. So, like a lot of folks, I started off really playing around with computers, goofing off with those when I was a little kid. And, you know, into my teen years, my dad had built a computer that my mom was using to write her doctoral thesis. And, you know, to be honest, it's kind of weird. At first, when I graduated from high school, it wasn't quite sure what I wanted to do. So I studied a couple of different things. I studied photography. I studied architecture, but eventually, you know, web-backed computers very happily, so. And then, you know, worked for a variety of companies. I first started off in banking and finance. So, we're full-walled as Fargo, Charles Schwab. And then, made the jump over to the agency world into consulting. So, I worked for a series of agencies ever since then. So, starting with Fraser Phage, I worked with a great company called Mechanism, where I was the head of engineering. And then, ultimately, I ended up, you know, from there, I went over to Edelman. And then, about 50, oh, not quite 15 years ago, about 12 years ago, I started working for a company called Presence. So, we built, as we like to say, we were building digital products for folks. So, you know, we'd work on websites and mobile applications. We worked on physical devices. You know, we did some IoT stuff. Very, very, very cool work. And while I was there, I took over as not only our head of engineering, but I also became our sister. So, really started driving, you know, what do we do with security? How do we think about securing our work, our clients' work, making sure everything is nice and healthy? And obviously, LLM's and AI came in along the way. Although, you know, AI's have been around a lot longer than LLM, right? Yeah. And about three years ago, we were bought by a company called Work & Co. And, you know, Presence completely merged up with Work & Co. And, you know, I'm obviously no longer leading all technology inside of Work & Co. It's a large organization, but I still play a role as a project lead, a delivery lead. And I also, in response to all of our security practices as well. I can be able to create any app you need in just minutes. Without writing the single line of code. With app for slap, that's exactly what you can do. Let's say you want to keep track of your expenses. All you have to do is grab an expense tracker template and you're ready to go in a few seconds. Now, let's see how this app works. I'll add a new expense category called Car Blow. You can create as many categories as you want. Groceries, travel, dining, whatever fits your lifestyle. Next, I'll record a quick transaction. August payment, $500, simple as that. And when I go back to the dashboard instantly, my total spending updates, everything is organized, clear, and ready whenever I need it. And that is the best part. It's just one example. With app for your lab platform, you can build apps for any of your needs, from fitness tracker, the project planner, to habit builders. Sign up free today and start building the apps you actually need. OK, great, thanks for sharing. I think this is good so people know about your work. So you mentioned AI, and this is the first something I want to talk to you about is with current new tools, AI tools like Copiala and such. How do you think those tools are changing how software development works these days? I mean, I think it's having a lot of impacts. I think some of it is for the best, and some of it is a little concerning. I think on the plus side, I think the AI tools like Copiala, Cloud, whatever, they bring a-- they are a very powerful tool for a knowledgeable developer. And I think that's the important thing, so that a knowledgeable developer can look at these tools and recognize that they can accelerate their work if they're used properly. So for example, writing boilerplate code, or writing unit tests, those kinds of things I think are really appropriate. I think where-- and that's positive, right? That's absolutely positive, right? So that you're getting an acceleration. It's like having your own dedicated junior developer by your site all day long. And I think for senior developer, that's amazingly powerful. I think the problems I'm having with it, where I think it's kind of making some changes, I think, not necessarily positive, changes are some folks, the vibe coding is the best example of this. I actually hate the term, I hate vibe coding, I hate the idea of it. And the reason is, is that you can go to an LLM, you can get it to crank out code, do whatever you want it to do. But unless you are an experienced developer, you're not going to know if the quality is there, right? You're not going to know if there's any security risks inside of that code. You're not going to know where the bugs are. And if you are a lazy developer and you have the LLM crank out all of your code, you're not going to understand it when it comes time to debug it. And so I think knowing those limitations is really important so that you can really take advantage of those strengths. Yeah, I agree. This is a very high area right now and a lot of concern. And since you're a security expert, and I also want to follow up here with the rise of vibe coding and all these people who are not engineers really. Like me, for example, if I go develop some application, acquire a bunch of users, that's probably a concern. So me as a consumer in the future, what should I be looking for, like a quality of the software, a quality of a product, how to assess that, what concerns we may have in the future with that low-quality products available out there? Do you see this at the risk? Yeah, I mean, I think for you as a consumer, there may not-- well, I want to be careful. So the code that's written for you as a consumer, there may not be as many risks, right? You're not going to be able to see the pre-compiled code and what's underneath the hood. Where it might impact you kind of indirectly is AI doesn't necessarily solve new problems, right? At the end of the day, what does AI do? In LLM, it goes back and it looks at all of the tokens, if you will, all of the content that's already been generated and uses that to generate new content, right? So most code, honestly, we are trying to solve existing problems, right? So it could be a very powerful tool there. But when you're trying to solve a new problem, you really have to have someone who understands the code and get it under the hoods and write it and solve your specific business problem. And where that might impact you as a consumer is, the code might be of a lower quality, it might be buggy, it might be very slow. It could potentially, if you don't really look at your code, the person kind of generating the code from an LLM, I was just in a form today where someone was talking about the fact that they were vibe coding. They asked the LLM to crank out a function and they kept trying to install, try to run that code and then install the library. And they learned that that library didn't exist, right? The LLM had just created the library and said, now you need to install this. And where that becomes a problem for a development team or for you as a consumer is, there's a very interesting security attack right now called Slopjacket, where hostile actors, very bad people discover that, hey, there is a library that's being hallucinated by an LLM, people are trying to download it, I'll go and create that library. And then I'll put my malicious code payload into that library, people download it, they'll run it, now I've owned them, right? So from that standpoint, that's a big risk for you as a consumer. The other stuff that I'd be worried about for you as a consumer is hallucinations, right? So hallucinations are part of an LLM. They're never going to go away. There's things we can do to kind of minimize the impact of them, but that's kind of the nature of them, right? Where it's it has to, there's some randomness that's generated into the output. That's why you always get different output when you ask the same question, right? And that's important, we definitely want that kind of variety. But you don't necessarily want to be careful when you're consuming stuff from a chat GPT or something like that or an application that has a heavy LLM component to make sure that hallucinations are accounted for. You also want to be aware of what's happening to your data when you're sharing it with an LLM, right? So is the company that, are they capturing your inputs? Are they using your inputs to train their system, their underlying system? And when they're capturing your data, are they securing it properly, right? So is it possible that your data could be used as a response to someone else's question? And if you're asking for, you're talking about your favorite restaurant, who cares? But if you're using this LLM or this system to deal with important critical business issues, family health issues, something like that, that's probably something you don't want dropped into a database that might eventually be exposed to the outside world. Yeah, that makes sense. I think it's important for people to know these risks. And I guess people get exposed to this more and more, they will learn over time, like pretty much everybody learned already to verify what chat GPT gives you back as a response, right? You cannot trust to anything. So. Well, I mean, just real quick, coming on that is I don't think it's up to the consumer to always make sure the tool that they're working, or that they're using, is working properly, right? Like, I expect the tool manufacturer to give you a tool that has been tested and safe and is ready to go. And I think one of my favorite usages of AI, I think one of the most powerful tools is the camera in the iPhone, right? Because it does a lot of computational photography. And, you know, it is an amazing, powerful tool. They've tested it. It doesn't do anything that's going to harm you as consumer. You can trust it. Occasionally, they'll be weird artifacts. You might see those like, you know, if you're taking a picture under low light. But at the end of the day, Apple has tested that. They put their name behind it and they put it out there, right? I expect that kind of behavior from any one of these AI producers, right? Be a machine learning, image recognition, LLM's, you know, vector search, all of that stuff. I really want that to be the responsibility of the producer not you as the consumer. Yeah, yeah. Are there any sort of misconceptions about AI and using software development that you heard about and maybe you can comment on? I think the, for me, the biggest one is that you can just solve a new problem with an LLM. If that problem, it does not exist in its training set, it can't solve it for you. And I think there are, you know, there've been a number of studies that demonstrate that. I think another one is, like I said, it's a vibe coding is that you can just have it write a ton of code and, you know, one person will be the next billion dollar company. You know, that's not possible. I, having said that, I think it's a bummer that people don't use it, it kind of lean into its strengths. It's a powerful prototyping tool. It's a powerful tool to challenge your assumptions. There was just an article on ThoughtWorks blog about, oh, gosh, a week ago, where I can remember the exact title, but I love this use case, where the person came in, they wrote unit tests, they ran through a bunch of examples, and they got to the end, they generated the use LLM to generate all their code, and then they rewrote it all. But because they had used the LLM, because they leaned into its strengths, they were able to write that function as a human, they knew what they were going after. They were able to write something that was maintainable, that was powerful, that was highly performant, because they use the LLM to get there, right? And so I love that idea. I think the title was something like, the best usage of an LLM is throw away the code or something like that. But it's a glib comet, but at the end of the day, they really leaned into the strengths of the LLM, and I think that's the way people should think about it, and not worry about the hype in the other places. Yeah, I think that's really the use case for wipe code in a similar approach, is just to sketch something really quick, demonstrate it, confirm it looks good, and then go rewrite it in a proper way. In fact, I went to meet up a few months ago for product managers, and most of them using these tools for exactly that. They want to sketch something up really quick, show to the developers, show to users, and then move on to the real stuff. So yeah, I think that's good. If you're enjoying this episode, it would mean a lot if you like, subscribe, or leave a quick comment. It really helps more people discover the show, explore these ideas, and be part of the conversation. Your support makes it all possible, so thank you for being here. And then also, from your perspective, when does it make sense to use those AI tools, and when does it make sense to stick with humans building from scratch, if you will? Well, I mean, I don't think there's a hard and fast roll at that point, right? It's like any other tool, you know? You, if you know the strengths and limitations of the tool, then it's up to you to make that choice. And so for me, it still gets back to the broad brush strokes, right, where it's like, you want to use the tool and drive it, and not kind of just fire and forget. That to me is really kind of the key. But I think from a development standpoint, like if you know what the tool is doing, and you know how to work with it, I am very happy to see a developer just, you know, take it to its full capabilities at that point. I see. And then, what are some of the concerns, specifically in security and maintenance of the code with AI tools like Copilot and Claude or whatnot? I mean, I think the big thing is, security is definitely the biggest issue. And I think it's, you have to go into with the assumption that it's not going to have the perfect security, you know, when you generate the code. It's like, I can't think of a prompt that says, write secure code, right? And so you have to treat any code written by an LLM as if it's code written by a very, very enthusiastic, but maybe not the most knowledgeable developer in the world, right? So if we go back to the way that we would treat code that was cut and paste from, you know, I don't know, like from any one of those, any sites, you know, that where you might go looking for code like Reddit or something like that, right? You still want to vet that stuff, right? So you want to go ahead and run it through some kind of static analysis, dynamic analysis, you want a human being to take a look at it, you want to do vulnerability testing, you maybe later on do a pen testing, right? You know, when you're done with that, you want to treat that code that comes out of an LLM the same way. You don't assume it's good. You make sure it goes through the static analysis. You make sure you have a person that's kind of taking a look at it. You don't assume that the libraries that it wants you to install are the libraries, are good libraries for that to exist. Another thing is you want to be very careful about the control that an LLM has over your desktop, right? So, you know, I've read horror stories about people, you know, again, I think it was just about two weeks ago where this person had an agent and they're working with the tool and it deleted their production database, right? And they couldn't recover it. And I've, you know, I read stories of people who are running the agents on the local machine and they let the agent do whatever it wants. And it now is installing malicious software on their computer, right? So, you really want to make sure that you are very cognizant of what you're allowing it to do and you want to make sure you have the ability to roll back, right? So, don't give it excessive permissions and then don't trust that it's going to do everything perfectly, right? The thing that's amazing to me, as I think about that is like, there's so many people who would never allow a human being to just deploy code willing in the production and just give a random human being complete control over their production database. So why would you do that with, you know, an LLM? You should just assume it's a hyper-enthusiastic but not very well-educated developer and go from there and go being a good shape. Yeah, yeah, I'm laughing because I kind of got to learn myself with this. I was experimenting a little bit with codecs from OpenAI and I'm not a software developer. I'm a IT guy and it's just like the name and that it provides the code, it generates the things it puts there and I have somebody who's really experienced looking over my shoulder and he always laughs and says, like, "No, no, that's junk, that doesn't look good." So, yeah, I think that's a very good analogy of having somebody who you don't trust, letting your code same with AI people should be cautious about that. Yeah. How do you think these tools, if people use them, development teams use them? Do you think it will allow them to spend more time on like design and architecture and be more thorough with how the product works versus just worrying about too much how to syntax and how code looks like and the repository organized? What do you think about like the organizational side of the code and how main table it, how much more readable it is and so forth? Yeah, I mean, I think it can. It can give you all that stuff. I think it does have the ability or the possibility of the potentiality of folks having more time, like you said, for the higher level stuff, architecture, you know, testing, thinking through security problems. So, yeah, I can totally see that. I mean, if it's writing boilerplate code for you, that's going to accelerate you, right? I think where it becomes more interesting is as you get into deeper and deeper, more complex situations, you might have to spend more and more time coming back and rewriting that code. And then, you know, if you're not very thoughtful and not very careful about it, you could end up with like the same function written 10 different times in your code base, right? And so, you know, you have to be very thoughtful about that to make sure that you're not losing, you know, the benefits of code reuse. So just kind of being cognizant of that. But yeah, I think it, I do know teams where it has been beneficial for them. They do feel like they have more time. But they've also recognized, again, that it's a tool that accelerates them. It's not a tool that replaces actual, you know, work of life and beings. Yeah, I think it's good to have some sort of like, engineer in one, one course for non-engineer people to kind of think through this thing. Because unless you have experience with it, you're not going to know what to look for. Like you mentioned, same function multiple times. Like I wouldn't know that. I wouldn't care even. So, but yeah. So you mentioned junior developers earlier. Do you think the company should like stop hiring junior developers? Because we have AI tools who do have some sort of opinion how to still leverage young developers. And they can still be productive and helpful for the organizations. I think we should be hiring junior developers. Our organization continues to do so. But not everyone can afford to do that. My worry is, you know, I know that the junior developers are finding it harder and harder to find work right now. And, you know, I think that's obviously that we'll have an impact on the bottom line for companies, you know, for the next, you know, I don't know, 5, 10 years. Lots of them. But in 5, 10 years, the senior developers are going to start to retire. And there will be no junior developers who've been trained to replace those senior developers. And so I don't have an answer for this, right? I hope that, like us, other companies will continue to hire junior developers and train them up. I worry that we won't have enough of those people. And we're going to have a very interesting challenge 10, 15, 20 years from now. Yeah, yeah, that's-- I'm not sure what we can do about this. Maybe some people have ideas. One I can think of is doing something on your own, right? Like a side project and just expand your skills. Maybe you have some other suggestions for young people who just graduate, you know, start in careers, how to train. If the company doesn't provide them training, for example, what can they do? Well, I mean, a couple of things. I mean, one-- I guess I do have one response that I think when you're doing your own projects, you're not going to learn how enterprise architectures work. You're not going to understand, you know, it's one thing if you have a small site that has to deal with 50 users. But what happens when you have to deal with 500 users, 5,000 users, 5,000,000 users. And you get that experience from working on those large-scale systems. And you're not going to be able to self-fund unless you've got that company on your own. I think a couple of things that can be done-- you know, there are some open source projects that, where some of the maintainers have some generosity. With their time, they might have a large number of maintainers. And so people will kind of build up a community, you know, where they can help folks. You know, they recognize, in their code base, everyone in a lot of ways is junior, right? So if you want to learn, you know, some complex code, you know, and you want to be abused. You know, eventually look at some of those larger projects. But you'll learn something, right? You know, you'll see what people are doing. You'll see the conversations. You'll figure out what people are trying to do there. I think there's just so many resources on the web. You know, so if there's a particular technology that you're interested in kind of growing in, I think, you know, doing a search, you know, looking for those resources kind of growing there, I think this is really important. There's certifications to pay upon the skills that you're looking for that I think are pretty valuable. So, you know, if you're very interested in DevOps, if you're interested in understanding infrastructure, you know, you can get AWS certifications, again, with a lot of materials out there to help you grow. And then, ideally, end up with a company, right? Where, you know, when you're interviewing, you know, unless you're desperate and you just need to take the job, but if you are interviewing, you know, my advice to junior developers is remember, you are interviewing that company as much as they are interviewing you. And so as a junior developer, you want to ask them questions about their culture. You want to ask them, you know, what are their learning opportunities? Do they have a budget available for training? Do they have a mentorship program? Do they do parent programming at times? Like, what's the collaborative environment look like? You know, and you want to make sure that you're ending up in a company, regardless of the size, that is going to be there and help you grow. And the smart companies, you know, I've seen even startups, and even today, I've seen some small startups, we're like, we've got a couple senior developers, we've got a couple junior developers, we'll bring them in. And this is our team for the next decade, right? They're already thinking ahead, right? They're thinking when the senior developers have moved on, our junior developers now know our code base and now there are senior developers. So look for those opportunities as well. - Yeah, yeah, that's a very good advice. So I also wanted to talk to you about like engineering management. And first thing I wanted to ask you is like, from your opinion, what are the responsibilities, like qualities of a good engineer and manager? - So I think as an engineering manager, it's your responsibility to get the hell out of the way. So and make sure there are no roadblocks for an engineering team, right? So that may sound like I'm saying just hands off, leave them alone and that's absolutely not the case, right? What it does mean though is you don't want to micromanage. You've hired smart people, you should have surrounded yourself with people who are smarter than you. And you should surrounded yourself with people you trust. So obviously you want to check in with them regularly. But once you've done that, you know, regularly maybe once a week, once every two weeks, whatever's the right rhythm for the person you're checking in with. And there's not a right answer for everyone. Some people want to talk to their manager on a daily basis. Some folks only want to talk to them like every two or three weeks. And you know, I do think you do need to check them with them, but you don't have to force it, right? Once you've checked in, you make sure they're feeling good, let them do their thing, right? So give folks space to do what they need to do. The second thing you need to do as a manager, is you need to manage outside of your team, right? So if your engineering manager or large organization, you should be taking care of all of the BS that can bog down your team, right? So talk to HR, you know, try to help, you know, get training resources for them if you're in a large enough organization. Make sure you're going to those larger meetings and then disseminating information to your team so they're kept up to base. If they need tools, make sure you're going out there and getting those tools for them. And then I would say the last two things is one, if your team is large enough, you're at some point as an engineering manager enough to make a choice. And it's going, the choice is going to be, do I want to keep writing code? Or do I want to lead the team, right? At some point, you know, if you are in the critical path and you're writing code 75% of the time, you can't do everything else to keep your team successful. So you should own that, be ready to walk away from the code and leverage the fact that you still understand how to program when you're talking to your team. And the last little piece is always have that open door, always be available to your team. And when your team comes to you, you want to listen. Don't decide that you have to take an action because sometimes that's all they want. They just want to let us some steam. They want to talk a problem through with you. Sometimes they're going to solve the problem on their own just by talking about. And you should be ready to be like, great. You know, I did my job. Oh, you solved your problem, great. Or, you know, they have a problem with someone else on the design team or the QA team or whatever. They just want to get out of their chest. That's not an indication that you now need to go and take an action. So listen to your team, ask them what they need, trust them and move on. - Yeah, that's great. I like those ideas or actions rather, how to be a good manager. It's sometimes especially with new managers, what I see as the problem is just, they just don't want to let go and continue writing code and continuing being in the weeds and completely forget like you said, those outside of the team issues that need to be solved. I think it's a good reminder for everybody to like, you have to shift your focus now. And then also the psychological safety has been quite a concern lately around companies. So how do you think a good manager should be paying attention to that and also paying attention to kind of have high performance from the team, accountability on the team and things like that? - Well, I mean, I'd start with the accountability. If you are going to hold your team accountable for certain issues, tasks, projects, responsibilities, whatever, you have to share those responsibilities, tasks, issues very clearly with your team. You need to set those expectations, okay? And then you need to be giving feedback when it's appropriate and on a regular basis. If you are doing an annual review, let's say you're a company that does an annual review and the first time the direct report is hearing about an issue is in that annual review than you have failed, right? There should never be any surprises when there's a review. When an issue comes up, you wanna share that with that person, right? But it all starts with, again, making sure they have a very clear understanding of what you expect of them. You wanna communicate that 100%. And those expectations are gonna change from sprint to sprint, right? You've assigned work out to your team. So you wanna make sure that they understand what that work is, that it's well-defined, and that you guys are all working together. The other thing is I really, really value the daily standup, particularly for remote teams, okay? And the reason is is that that 15 minutes, and I like a 15 minutes standup, by the way. I don't like this idea of like, we're gonna have a standup, it's an hour later, we've covered all the things. No, you're getting, you do the standup, and you let people, this is what I did yesterday, this is what I'm doing today, these are my blockers. Take note of that stuff, right? Like you as the manager of that team, you know, you should be cognizant of what people worked on yesterday, what they're doing today. And if you see the same issues kind of showing up a couple days in a row, that's a time to step in, right? Hey, you know, let's kind of go off to this side, Vin, or vid, I wanna, I have a couple of conversations, or I have a couple of questions for you, I wanna make sure that, you know, you're doing good, it sounds like this issue is kind of taking a little longer than you expected, you have a blocker, how can it help you out? Right? You know, I think if you're not as a manager, if you're not having those opportunities to check in with your whole team on a daily basis, you're not gonna catch that rhythm of the team. And part of the other place you can check in, is in Slack, right? You should see conversations, you should be lively, people should be talking about stuff, you should, you know, see people jumping in and helping each other, you should see folks asking for pull request reviews, stuff like that. If you've got a really quiet Slack space, that's a red flag. You should start trying to figure out why your team is collaborating together. - Yeah, that's what a good point collaboration. I think I serve as a scrum master for a couple of teams, and I sometimes try to encourage people to do this, like communicate, don't wait for the next day, for the next standup, just like somebody will pin them one on one and things like that. I think it's very important to keep motivating people to collaborate and things like that. Yeah. - Yeah, and I think it's up to you as a leader, right, to set that example, you know, so if you schedule, if you for whatever reason are scheduled to be in the standup or you schedule the standup, depending on the size of your team, you should show up regularly. You know, if you've got a large organization that you're managing clearly, you're not gonna be able to go to every standup every day, but popping in, checking in on the team, talking to the leaders of the folks on your team is really important kind of getting that rhythm. Every once in a while, you know, talking to, you know, the folks that report to your direct reports, they think it's powerful. If you're on a smaller team, you know, you might be the engineering lead as well as the scrum master, right? So you absolutely should be there on a daily basis and listening in. If you're not showing up, the team's gonna think, well, it's not important, they're not gonna show up, right? You set the example. You know, you shouldn't have to tell people what the expectations are, you should be able to show them what the expectations are. - Yeah, yeah, that's great. I thought I'd agree with that. And then another thing is, I think, important for managers to help teams with, is burnout with their new tools, like people using AI frustrating, maybe sometimes. They also have to deal with technical debt. We know how painful it is sometimes and people always hate to deal with it. And any other various things, like leadership pushing like can release this fast and fast, because we have some sort of advice from managers, how to deal with that and keep their teams motivated, I guess. And deal with this, not working out. - I mean, kind of addressing that first one, which is like that idea of, you know, you're getting pressure from outside of the development organization to constantly deliver. And the, you know, the pace is untenable. It's like, it's crushing, right? I think you, this is that moment where you use your manager, you kind of figure out what you're about, who you are. And you have to be willing to go to the rest of the organization and say, look, I'm going to burn out this team. They're going to leave. Our velocity is going to go down. The quality is going to go down. We're going to get a bad reputation in the industry, right? We need to, we need to be a lot more thoughtful about the tasks, the pace, the pressures that we're putting the team under, right? Every organization is going to have some crunch time. Like I recognize, we all recognize that, right? There's a deliverable, we can't miss it. That two weeks is a death march, right? You're just pushing through it. If every sprint is a death march, like if you have 26 death marches a year, you have an organization problem that you as an engineering manager need to address. And you shouldn't wait until it's like six sprints in and everyone's been working 80 hour weeks for 12 weeks. You're going to see that on Sprint 1 or Sprint 2, right? So you've got to step up and take care of that. Another thing as a manager is you've got to encourage people to take their time off, take their vacation. So if your organization offers two weeks of vacation, three weeks of vacation, you should be aware which of the folks on your team have and haven't taken vacation time. You can't force people to take vacation, but you can encourage them and you should absolutely do that. The other thing is I work really odd hours. I take a long lunch, but I start my work day at 7 a.m., I finish about 5 p.m., right? So you think, wow, it's working a long day. But I take a long lunch, right? But a lot of the folks I work with are on these coasts. So when three o'clock rolls around, two o'clock rolls around, I don't want people to respond to me. I want people to get to their five o'clock, handle their work day, and I want them to check out. Now, if I'm sending them Slack messages, or I'm sending them emails at 8 o'clock their time, they're going to think they have to respond. And every tool, every communication tool nowadays, I'll almost solve them. I can't think of one that doesn't do this. You as a manager, if you want to avoid thinking your team can't take a rest, which they need to avoid burnout, then you need to schedule those messages after your team is offline, after they've gone home. Don't message them on the weekend. Don't message them at 8 o'clock, 9 o'clock at night. Don't message them when they're on vacation. In fact, when they're on vacation, you as a manager, you need to make sure that they can check out, that they don't have to bring their computer with them. So you need to make sure that their tasks, and their responsibilities, and their duties, have been taken over by the rest of the team. And if that means going to the leadership and the rest of your organization, say, look, we're going to be down by a person for two weeks. That means we're going to deliver less. We need to figure that out. Then you as a manager need to do that. If you can't do all that stuff, you're not going to be able to keep your came from burying out. Yeah, that's great to hear these things as a suggestions for people. Well, we have a policy and organization not for other people after hour, so weekends. In fact, we have strict policy about that, but some private companies may not have that. I remember my times and start up. And it was 24/7. Anytime somebody can pin you even clients, and you can't get out of this. So it's just terrible. But for the most part, I think it's good to protect personal time. Well, and I think even if your organization doesn't have formal rules about protecting people's time, again, you as that manager, you can make that choice. So again, you can set that example. If you're picking them at all hours of the day and night, they're going to start to think they need to respond. And so you're burning them out. Just don't. You can set your own policies. Yeah, that's right. So next one is, how do you suggest measuring team health? And what I mean by that is performance, how they do things, right? And also their human signals, like burning out would be one and things like that. So I'm reluctant to say this, because some folks will use it wrong. I think you can use story points as a way to see how the team is working, right? And so I want to be very careful about this, because I think a lot of organizations will use story points as a way of measuring performance. And that's not necessarily what they're there for, right? And so you want to be careful about the way that you as a lead are looking at them. However, you can use them as a way to see if there might be a problem. So let's say your team normally is doing, I don't know, picking number six, story points per person per string. And everything's jogging along. And then you have a couple of sprints where the numbers go down to like three story points. That says three story points per person per string. You can now go, OK, something's changed. As a manager, you can take that as an opportunity to kind of come in and take a look. Are we not writing stories current enough anymore? Are people over at the main amount of work they can do? Are they getting caught up in issues? And they're not kind of talking about them? So I think that's one thing. That's one way to see how the team is performing and checking in on the health. Another thing is to listen in those stand-ups. So if you have a developer who's normally pretty happy, go lucky, and really friendly, and offers to help everyone, and then they're suddenly like really snappy, or they're like, do it yourself. I don't have any time. And it might not even be that traumatic. Maybe they're just really quiet. Normally they're very effusive, very talkative, and they get very, very quiet. Listening in the stand-up, really listen, really pay attention. You'll start to get those audio signals, right? And then those are my two biggest ones. I would say the other thing is keeping an eye on things like bugs that might be generated, kind of, some bug reports. And what I mean by that, I want to be careful about that, too, right? There's no such thing as bug-free code. But if you suddenly find that your team is cranking out code, a ton of code, and it's getting a lot of bugs that are showing up, right? And maybe using a code quality to a, like, sonar cloud or something like that. And suddenly you're getting a lot of negative reports on that. That might be an indication that your team is feeling and pressured to deliver more than they safely can during the course of a spread. So those things are, I think, good indications of, like, potential quality issues. And I think potential quality issues are potentially indications of burnout. Yeah. OK. So we're almost at the end here. And I wanted to ask about some sort of advice for somebody who's maybe transitioning to leadership role, or already a leader, and something maybe you can share with them how to be a good leader. I'll go back to my biggest thing. I know it's, I know I'm repeating it, but I think it's the most important thing. Someone once said to me, I just, I respect this so much. He said he has two ears and one mouth. And he tries to use those in their proportion. And I think, I remember when I first transitioned into leadership, I felt like I had to be in every single conversation. And I had to look at every line of code. And I had to solve every problem. And I was driving my team crazy looking back on it. And I realized that the most important thing was for me to listen, to be there for the team, to make sure they knew my door was always open. Either the physical door or the virtual door. If my team ever pings me in Slack, it's like, you know, I turn off almost all notifications in Slack. I hate that knock knock. I don't hear it. I get very few of those, you know, my red circle on my Slack icon is usually like one, two, three digits, right? It's, you know, it's between one and 10. I don't have like 100 messages waiting for me. But I always make sure that if my people send me direct messages, I'll see that. And I'll respond right away. I think you need to have that kind of availability to your team. And obviously there may be times you may be in a meeting with management you need to do that. But otherwise, you need to make yourself available to your team. And you need to listen to your team. And don't try to force the solution or an action on them. Be there for them. Listen, have that open door. When you do that, they'll start to trust you. They'll start to approach you. And you'll be successful as a manager. Yeah, yeah, that's a great advice. And then the last one is, what are some of the excitements or maybe you cautious about, especially with AI, for the next few years, specifically in software development, Ada? You know, I don't want to leave everyone thinking that I'm a big, you know, curmudgeon when it comes to AI. I am pretty excited about some-- like I said, I think it can be a very powerful tool with boilerplate stuff. I really, really, really like what is happening with AI, with security stuff. So where you get tons of alerts and you can use AI to filter those down to the alerts that matter. So your team doesn't get alert fatigue. I really-- I think there's some powerful stuff there. And I'm looking forward to a lot of companies going out of business so that we can focus on the companies that are really good at supporting developers with powerful AI tools. That, to me, is going to be a great moment. And then we'll have some tools that will make everyone's life better, and I'm looking forward to that. Yeah, yeah, that's great. OK, Ada, thank you very much for our conversation. A lot of good information and wisdom from you. And I'm sure people will appreciate all of that. Thank you. This is my pleasure. All right. Thank you. Back to you later. Bye-bye. Bye. Thank you for tuning in. I hope this conversation gave you fresh ideas and inspiration. If so, please share it with your friends and colleagues. Don't forget to follow us, subscribe, and leave a quick comment. It helps more people discover this episode. See you next time.

Podcast Summary

Key Points:

  1. AI tools like Copilot and Claude can accelerate development for experienced engineers by handling boilerplate code and unit tests, but they require oversight to ensure quality and security.
  2. Risks include "vibe coding" by inexperienced users leading to buggy, insecure software; hallucinations in generated content; and security vulnerabilities like "slopjacket" attacks where malicious libraries are created to exploit AI-generated code.
  3. AI is best used as a prototyping or brainstorming tool, not as a replacement for human problem-solving, especially for novel challenges not present in training data.
  4. Developers must treat AI-generated code with caution—vetting it through static analysis, human review, and limiting permissions to prevent incidents like unauthorized database deletions.
  5. The responsibility for safe, reliable AI tools lies with producers, not consumers, emphasizing the need for rigorous testing and transparency in AI systems.

Summary:

In this podcast interview with Eric Miller, an engineering leader and security expert, the discussion centers on how AI is transforming software development. AI tools such as Copilot and Claude offer significant benefits by accelerating tasks like writing boilerplate code and generating unit tests, effectively acting as a "junior developer" for experienced engineers. However, risks arise with "vibe coding," where inexperienced users rely too heavily on AI, potentially producing low-quality, insecure software.

Key concerns include AI hallucinations, security vulnerabilities like "slopjacket" attacks, and over-trust in generated code without proper vetting. Miller emphasizes that AI excels as a prototyping tool but cannot solve novel problems absent from its training data. He advises treating AI-generated code with skepticism, subjecting it to static analysis, human review, and strict permission controls to avoid security breaches.

Ultimately, the responsibility for building safe AI systems rests with producers, not consumers, and healthy engineering teams should leverage AI to enhance—not replace—human expertise, focusing on higher-level design and architecture.

FAQs

AI tools can accelerate development by generating boilerplate code and unit tests, acting like a dedicated junior developer for experienced engineers. They allow senior developers to focus on higher-level tasks like architecture and security.

Vibe coding can produce low-quality, buggy, or insecure code if the user lacks experience to evaluate it. It may also lead to hallucinations, such as non-existent libraries, creating security vulnerabilities like supply chain attacks.

AI-generated code should be treated as written by an enthusiastic but potentially unknowledgeable developer. It requires vetting through static analysis, human review, and vulnerability testing to avoid security risks like malicious libraries or excessive permissions.

Consumers should look for signs of buggy or slow performance, and be cautious of applications with heavy LLM components due to potential hallucinations. They should also verify data privacy practices of the provider.

Use AI as a prototyping tool or to challenge assumptions, but rewrite the code for maintainability and performance. Always understand the tool's limitations and ensure human oversight, especially for security and architecture.

A common misconception is that AI can solve entirely new problems not in its training data. Another is that it can replace human developers entirely, rather than augmenting their work when used thoughtfully.

Chat with AI

Loading...

Pro features

Go deeper with this episode

Unlock creator-grade tools that turn any transcript into show notes and subtitle files.