Hands-On IT – A Prompt Engineering Deep Dive with Henry Smith, E20

All right, hello everyone and welcome back to the auto mox hands on it podcast. We've got a really exciting show today we're going to be talking about a i and prompt engineering and the special guest with me today you may recognize him from the patch Tuesday podcast where he is a frequent host on that also but Henry Smith who is a security engineer at auto mox will be joining us today. [Music] Henry you want to introduce yourself real quick. Sure, yeah my name is Henry as Landon said security engineer here at auto mox. I always struggle with these introductions I'm like what do I say I do because it kind of changes like all the time but for the most part you know just here working on the security team and trying to build our security program and you know make it make it better every single day so yeah that's one of the best parts of working at auto mox I think it's like someone like hey what do you do is like well today was different yesterday which has been different than every other day so yeah I should clarify. I should clarify I love it I love it too I get to I don't know I always joke that I explain complex things to people and that seems to go forward well enough but yeah so but it's it's been a great place to work how long have you been in auto mox and what's your what's your background how did you get to auto mox will do that I always like to. I always think that's interesting people do that on podcast I'm going to do that on my podcast so sure yeah so gosh if I'm my memory serves me well I think it's been two years maybe over two years at this point it's been a wild roller coaster as it was my first my first startup so yeah anyone that's worked out a startup knows what's like so I had a lot of growing pains to do there but. I figured what your other question was. Oh how did you how did you get here and what did you what kind of what was your path to getting to auto mox whether through college or kind of has this always have you always been in the cyber security industry. Alright I every time someone asked for this I feel like I always talk way too long so I'm going to try to keep this short long story short you know I started out growing up as a huge computer nerd and loved it information technology got my first it job pretty much right out of high school I tried college it wasn't for me so I don't have a degree so I kind of just have always been career driven worked for a couple MSPs throughout my time. And eventually at one of the MSPs I've bit by this by the security bug as I like to say and ever since then I you know I took my first my first security course got a certification and kind of everything after that. It just kind of exploded from there and I ended up my first security job actually ended up back at a previous job where I was a technical support engineer. And they they brought me in as kind of like a you know an entry level security engineer and I've ever said that it's kind of been my journey and here I am. Yeah that's I think that's actually what we're talking about and next month's podcast to is kind of career paths and you know college really isn't for everyone and I tried college for a while too. And then it started working and ended up going back to college shout out Arizona state actually have never been on campus but I went online and for some reason that's what worked for my brain is being able to fast forward and rewind lectures instead of having to sit still in a sit still in a classroom for a while. You know it only took me let's see 11 years to graduate college but hey you know what I did it. So yeah between the first try and then the second try and yeah it's like yeah which will go into this in another podcast getting a little off topic too but just go for what's your my career advice go for what you're good at not for what you think you should be doing. And what you love to yeah and just one little piece networking because I really I truly don't think that I would be where I am today I don't even think I would have my first security job if I didn't already have connections at that company. Yeah it's really it's so important to network but I digress like yeah. Yeah the other thing we talk about a lot on this podcast is that most of us are in IT or security because computers are the easy part but it's the intersection of people and computers is what we do but just being able to talk to people and not completely overwhelmed them with technical information and the first like it's like hey I'm going to come fix your computer I'm going to go through this registry key but just being like hey like I'm not sure what's going on here we're going to get you fixed up though. It's a lot a lot better just having those people skills and networking skills is is critically important like you said but off topic a little bit come back next month if you want to hear more about this because that's what we're going to be talking about because almost everyone at auto mocks that I work with has had kind of the non standard career journey whether it's Ryan who was in a rock band and got his degree in the back of a van on tour or Henry or anybody out here. Tom and Jason who were in the military for a long time and then went into cyber command and stuff but it's always there's there's no one right way to go after your career and that's that's one thing I love about the IT industry is that if you're you're good at it you can prove yourself and you don't have to be like look at this piece of paper I have. So right look at all this money that I've spent I mean no fun to anyone that I want to say that like I really I think continuing education and college is important if it's the right thing for you. Yeah and I mean it's not like I it was I'm not good at sitting down and doing things for a long period of time and like especially sitting through a lecture like yeah not going to lie I think I slept through most of my lectures the first time I went to college but tell you what if you go back to colleges and adult and actually do the homework and listen to the lectures it's a lot easier. But yeah anyways I digress back to back to AI back to AI so it's going to no one's going to need college here but anyways we'll go. Alright prompt engineering so prompt engineering think I like what is it I like to think of it kind of like talking to my kids or a young kid like you talk to your kids differently than I would talk to Henry or someone else that's highly technical so the way we talk to these AI systems matters and it can shape the output. Henry kind of you want to add anything on that what are your what are your thoughts on prompts engineering. I would say yes I agree with what you said and I would tack on that it's like good prompt engineering is like the difference between working with someone that's on their first day at the job and someone that's been there for a while. I have a training they have context to do the job in a way that follows conventions and standards that are already in place so yes it's important. Absolutely so I always we were talking about this a little bit before the show and I always kind of think of it we used to read these books when I was in elementary school there were Amelia Bidelia and she took everything very literally so it's like hey she was a maid and it was like hey we go draw the curtains and she like gets out of notebook and draws a picture of the curtains and. I kind of think of prompting AI is similar to that where it either can take things literally or in the wrong direction and cause a lot more work for you. That's kind of how I like to think about is like okay like if I am describing what I want the constraints that I need everything you need will go that way and. The common common guide a lot of times has been like treat AI as your junior developer give it as much context as possible so that's a great way I don't 100% agree with this my theory is to kind of prompt the AI as if someone was asking you to do something and what you would want instead of like hey let's go do this and I mean there's. There's kind of this breakdown of like hey AI understands what you're talking about a lot of times when we're explaining things to other people we kind of sugar coded or make it a little bit easier to understand it's like no like this it understands this technical term i'm going to use it's an industry standard one sometimes if it's technical term you made up it's not going to understand it but like. Being able to talk to it in a way that you would talk to yourself or push to another developer junior developer that's that's kind of the advice that I would give Henry kind of I'll let you I'll let you run on that one a little bit to is there anything. Yeah so kind of what you were saying if you if you asked someone to perform a task and you didn't give them any context what would they do chances are they would likely ask questions either to you or someone else to try and get more context or they would. Do some research to try to get that context themselves and I feel the AI agents especially are very similar in the job tends to get done faster and better if. the right context and information is there to accomplish that task. >> Yeah. And this goes for anything you're using an AI for. We're going to be mainly talking about creating and generating code with AI, but if you're trying to write a creative novel or whatever you're working on a blog post or anything, being able to give it the information that you want and have an idea of what you want the output to be is critical in making it work the way you want it to and not cause more work for you. >> Right. And I've also found that it's helpful to, at the end of your instructions, say, if you have clarifying questions, please. >> Yeah. >> So that way it doesn't assume things. Because back to your point earlier, if someone, if you're trying to explain something to someone and they don't know what it is, they also might try to make an assumption about what you're talking about. And I feel like, again, AI works very similarly. >> Yeah, I totally agree. And that, yeah, this is a lot of the more advanced models. Like I think, Chad G, open AI's O3 model. If you're doing deep research, if you prompt it, it will come back and say, hey, what about this, this, and this? What's direction do you want me to run? So it's getting better at asking questions, but being able to kind of guide it in the way that you want it to still critically important. And I mean, in a lot of times, you can use AI to get feedback on your prompts, too. And it's like, hey, like, I'm writing this prompt. This is what I'm thinking. What areas do I need to improve to make this more clear? >> Not. >> And it will come back with something. Sometimes it just starts the prompt and starts doing that, but I think that might be my problem, too. So, but yeah, that's just kind of being able to bounce that idea off of someone or off of an AI system, and LLM is great for that feedback loop to make your prompts better, make the outputs from these LLMs better, too. >> That's an LL. >> Yeah, and then the other thing that especially code related that I've used AI a lot for, and I think is a really good use of AI is explaining code sections. Henry, have you used it that we're like finding a bug or stuff, just like, hey, explain this code section for me. Why is it working this way, give me an outline? >> I've been about this for a long time, because I've been experimenting with it a lot. Yeah, so I've definitely used it for exactly that purpose. Say, hey, what is this code doing? What does it actually mean? What are these things talking to? How are they talking to each other? What are they saying? Just getting down to that nitty-gritty detail, especially if I haven't seen a code base before, after the first time I'm seeing it, I've found it is so helpful to just say, hey, tell me what you know about this. >> Yeah, yeah, absolutely. I think the AI model that I've been using a lot is Anthropics Cloud Code recently, and it can drop into a code base and you can say, hey, tell me about this code base. It can look at the file, go look at its subscripts or any references and give you a really good overview and understand that option too. Then, sorry, understand that code and give you an overview of what's going on with it and how it works. That's been incredibly useful to me in my job. The next area is generating code and auditing code and hardening code. I know that you've been doing a lot of experimentation with that lately, so you want to dive into those concerns and security concerns for code and AI-generated code. >> Yes, so I'll kind of take a step back a little bit and go back to the context piece. >> Yeah. >> Again, context really is king when it comes to working with LLMs. It really is the difference between working with a newbie and someone that's been there for a while. I've personally seen a significant difference in what something like cloud code will output when you give it context and when you don't. In the past few days, just here at Automux, I've been experimenting. For example, I had it create a backend for microservice and I gave it very generic instructions. Like, I want this thing and I want it to do xyz, please build it. I admit the results were okay. They were pretty impressive, but they weren't great. They didn't, the code didn't follow the kind of standards and conventions that we have here at Automux. It really just kind of assumed and did its best job. After that, I cleared out all the context around the conversation, so we had a fresh slate. Then I pulled in AI, or I pulled in AI agent instructions from a markdown file. With a bunch of information around our coding standards, both related to and unrelated to security and our conventions and really just context around how we engineer here at Automux. After I did that, I asked, "Claw the same question," but I was very specific and I said, "Use this file for context." And the output was astoundingly improved from the last iteration. Amazingly. I've been experimenting with this too for PR reviews. I found that it's not really a useful tool unless you provide it some sort of context that a human being would need themselves to be effective. If you were to ask, "Hey, co-worker, can you please review my code?" They're going to pull up your code and if they don't have any context around what you're working on, they're going to struggle to review the code. Yeah, it's the difference if you'd be like, "Hey, mom, can you review this code?" And they're being like, "It looks like code." I don't know anything about this versus like going to an experienced developer at your company that has the background in what your company is doing and saying, "Can you review this code?" And like, "Okay, I don't know. I like to think of it as put your mind in a markdown file. Everything you know about this that is specific to the company or the specific to the project you're working on, give it those instructions and then push forward." So letting it know what you're saying, letting it know what you're working on, the background of this, what it's supposed to be doing, how it interacts with other things can significantly change the output and the quality of your code. I like to often, my kind of my workflow lately with some of the stuff I've been working on is, it's like, "Okay, let's get something that works. It doesn't have to be the best, but I want this, sometimes I'll do it or have a wired diagram or something." It's like, "Okay, this works. It sends the data to the right spot. It sends it to this database. It looks at it." It's like, "Okay, this is how I want it to work. Optimize this. Make it, yeah, so telling it to optimize the code, telling it to harden the code, looking at these things, it will often, this is what I want it to work. This is how I want it to work. Okay, I understand that. It will take that and they're like, "Okay, well, we're going to harden this section. We're going to batch the right to the database on this section, so that it will speed it up." So you're not waiting 30 minutes for 20 records to be written or whatever it is. That's really slow database times. But just having it optimize things is that's how I've been using it a lot lately. This is what it should do. This is barely working, but it works. Make it work better. So on the security review side of things also, do you want to touch on that a little bit, too, or do you want to actually, I'll let you finish what you're saying about the code quality of prompting if you have anything else to add on that. Well, I feel like they kind of go hand in hand a little bit. So back to the little context thing in the agent AI instructions. So first of all, a lot of these agents will actually produce a file for you if you don't have one already. So, Claude is a great example. If you just do slash and knit, it will go through your code base. I'm pretty sure it'll even look at get logs and things like that, whatever it needs to build context and information around your code base, your repository. Yeah. And so I've been experimenting with that and what I've been doing for security things is I've been adding like directives to it. So it's just a little section in there that is specifically just bullet points. Just one or two cents in bullet points with like, you know, you will validate all user input. You know, things like that, you will validate that like a requester, you know, has proper access to the resource that they're trying to access. Just having those directives in their Claude, for example, can actually use that context in an automated PR review and it can say, hey, I know you have these code standards here and you have these secure coding standards. Yeah. And it looks like you're not following them here. What's up, man? And you can even go as far as as to categorize each of those directives to say, "This is a must and this is a should." So, when Claude's reviewing it'll say, "Hey, you have this as should, so I'm just going to put up a yellow bang, a warning here." Or, "If it's a critical warning, you can have it give severities to the things it finds, it's infinitely customizable." Yeah, and the other way I've been using it to sometimes is, "Hey, do a security audit. Is there any sections of this code segment right here that could be exploited that I could harden?" And oftentimes, it'll come back and say, "Hey, validate the input. The input validation right here can be beeped up a little bit. You can add a lot more to that." Or, "Look for any SQL injection problems here if you're writing to a database." And it'll say, "Okay, I can do that." I'm just like, "Let's do that." And my favorite is, I don't know, sometimes I've been working with Claude code and I was like, "Hey, I was trying to troubleshoot an issue of why something wasn't working correctly." And it's like, "Well, it looks like you did this and this isn't working." I was like, "Wait a second. You wrote this code, Claude." This is on you. I'm going to do that. Yeah. Go fix your stuff, man. Yeah, so that's been just kind of hardening and analyzing code. I mean, as humans were limited in the way that we can think about things and the way that we see things in our knowledge base, but being able to just go out to someone who someone has an AI system who can scan through it quickly and say, "Yes, this looks like it's up to your standards or no. This isn't up to your standards." I mean, it's not necessarily going to replace you. I mean, it still has to have that human oversight, but like it's an incredibly powerful tool for that. So. Yeah, and now that you touched on that, I want to say, I'm not advocating for no manual reviews and just rely on AI. But I am advocating for leverage AI to help to review those. It's kind of like vibe coding is the term that's come up a lot. And to me, that means, I'm going to have an AI build something and I'm going to have no idea how it works. And I'm going to hit him and ascend it. And we're going to see what happens. And don't do that. Especially in production, if you're just developing something to test out, that is fine. But if it's in production, don't do that. So, what kind of goes vibe coding versus assisted coding? Those are two different methods of doing things where I'm using AI as an assistant to make me more productive. I can review these things faster. I can sometimes find errors that I would overlook versus just saying, "Yep, AI said it's good. Let's send it." Yes, 100%. And again, just want to say, it all comes down to having that context in. Telling the AI, I want you to look for access control issues. I'm feeling like I'm beating a dead horse here, but I want to sell. I think this dead horse needs to be beat. Nost, like, context is king. If you want good results, you have to provide it. Yeah. And it's, yeah, it goes, it's so interesting to see, like, go experiment with this on your own time, but go write a really bad prompt and then spend like 10 minutes to write something really good and compare the outputs. And it's remarkably different. And I have a perfect example of this, internally. And I bet you we could maybe share if we were going to do a blog that supplements this. We might have to do a blog on this thing. Yeah, because I have a perfect example of that, where I took a screenshot of a prompt where I was like, "Make me an ice cream cone," and I didn't give it any context. And it made the most horrendous thing. And then giving it much better prompt, it actually output a ice cream cone that was very well put together. So I love it. I love it. And I mean, I think across any kind of AI also, whether that's cogeneration, image generation or anything like being able to give it context to what you want, being able to kind of know what you want before you go in there is critical. So. That's another thing too, especially if you're trying to be conscious of your token usage. Yeah. If that's another place where context and good prompt engineering helps to save you money, because if the AI has to make assumptions about what you want, and then you have to go back and fix them, you're going to spend more tokens than you want to be. Yeah. Yeah, absolutely. And so the other thing I wanted to chat about a little bit too is context window. So especially with token-based AI systems like cloud or or other things, it I mean, even with chat GBT or any of those systems, there's a context window. There's only so much information that it can process. So staying within that context window is important. I mean, so I think that the latest one is, I mean, it's getting much larger, but it's still something to be to be cognizant of. And like, it can only understand so much of my code base. Like, I'm not going to just say, here's this 10 million line code base, figure out everything about it and go fix all the problems. That's not how it works. Being able to get to the smaller sections that you want it to analyze and analyzing kind of more specific problems, it's really efficient at, but you can run out of context window. I mean, we've been talking about cloud a lot because that's what we've been using a lot. But even that one at the bottom of it will say, hey, you've got like 20% left until I'm going to start auto-compressing stuff. But just that context window is how much memory it can hold about what you're talking to it about in one spot. So you start to lose efficiencies if you're having to explain something multiple times. So like, if I explain this once really well, you can keep running with it and not have to ask more questions and that claps that window of memory. So yeah. And that is where the AI kind of instruction set that I was referring to really helps and comes into play. And if you can even centralize, so going back to thinking just like an engineering department, if you can go back, if you can centralize those AI instructions as well to be consumed, you know, that might help the context, the context window problem a little bit. And one thing I've been doing too, just personally experimenting is when I'm done with a session, I'll say, cloud, tell like right, basically know everything we did in this like context.md. It's marked on file and put everything in there. And then if I close the session and come back to it later, I'll say, hey, look at that. I see what you did last time. Yeah. And then it actually will help me fix this like new, this like other bug I found so much faster than it would if it had to re-understand the code. Everything. Yeah. And then you have to tell it the same things over and again. And if you don't, if another thing that you can experiment with to our listeners is if you have a chat GPD subscription, you can create a custom GPT. You can put that context window in there. And that's something that's kind of a lower entry way than necessarily trying to figure out the command light interface of cloud. But if you just kind of want to play with something, that's a really good area to play with. You can say use these instructions. This is my instructions. This is my style guide or this is what tone I want it to use. And you can practice kind of prompting it. If you get something that's really good in that, you can, that's being able to prompt it. You can go back and forth and test it and see how it reacts to having those instructions or not or modifying those instructions and how that works. So if you kind of want to practice with this and practice kind of building out those content windows or content instructions, that's really what I would would recommend kind of as a starting point if you just want to play with something is go build a custom GPT in open AI's chat GPT web platform. That's a fairly low barrier to entry. So I agree. I think that's good advice. And honestly, if you haven't been experimenting with AI, I think you really should start. Yeah. Absolutely. And it's flowing up. Yeah. Yeah. It's kind of like the internet. It's just a fad. It's funny though. There's almost no end in sight. You see the latest model comes out and it does. It promises all these amazing things that the other one doesn't. And then sometime later, oh, there's other models now. And it promises even more and it's getting better. And like, it's the velocity is just not stopped. It is. And like a lot of the, it's interesting because a lot of the promises from these AI companies like, yeah, no, I'm, it's not going to do that for me. But like, we can do this really well for me. It can help. It's like kind of cutting through the marketing stuff. and be like well I could use it for this. that sounds really interesting. And just reading through that, kind of coming up with ideas, how you could use it or play with it. And don't go spend $200 a month for whatever it is for open AI's top tier subscription. But you can do the custom GPDs, I think with the, I think it's with the plus 20 bucks a month or whatever the idea to pay them a little bit for it. But if you're really wanting to go play with it and learn AI and like you just go play with it. And like that's my advice is like see what you can see what you can do. Go have fun. So I have an echoed same thing. Yeah. Yeah, there's, I don't know, there's not too much that like I asked it to help me come up with a recipe. It's like I got this stuff in my fridge. I think what can you do? What can I do with this? And turned out really good. So, but I mean, there's a lot of a lot of stuff that you can just kind of experiment with. Have fun. Like go go play with it. See what you can see what you can do. And that's a, I don't know, it's kind of my ethos with anything that I find interesting. It's like, well, how can I play with this? So yes. Hands on learning is my also my approach too. I just like, I gotta try. I gotta try. I gotta try. I gotta push all the buttons and see what happens. Right. And I need to get burned or like, I need to, yeah. Yeah. Yeah. I just worked out very well. This worked out good. It was fun. Like, but yeah, it's, yeah, go play with it. That's, that's our advice. I guess. But, but yeah, also, Henry wrote a really interesting blog a couple of weeks ago. Actually, earlier this week, I believe, that is on about a Splunk integration with AutoMox. So go check that out. He did a great job writing that one and kind of for more of stuff that Henry's been working on. But, but yeah, is there anything else you've been working on AI wise and security wise that you want to share? I think I kind of shared everything there. You know, just we actually started a working group here at AutoMox. Yeah. And so part of that is really bad. Everything I said earlier, I think I talked about earlier with this experimenting and really, really trying to think about, you know, our developers and like the development process. And we want it, like we want them to use the tool. But we want to make it as easy as possible to adopt in the safest way possible. So it's, I'm trying to find that balance. So I would say that's one thing I've been working on. Yeah. And it's, I will say that it's one of the most interesting things about AI to me is seeing how other people use it. It's like, wow, that's a really interesting way. I've never would have thought of that or like, and it's just kind of this back and forth. And then you share something that you've been doing and they have the same reaction. And it's this kind of, it's a yeah, talk with find someone else that's using AI a lot. Talk with them, see how they're using it. You'll be, you'll be impressed. It's, there's a lot of fun things you can do with it. And it never gets tired of answering your weird questions or answering, building strange things for you or it's a bouncing ideas off of is it's kind of like the, I don't know, like when your kid is like, Hey, I've got this question. I've got this question. And you're like, I don't know the answer to any of these. But with a AI, you can just keep going with all sorts of random rabbits. Like the ultimate Wikipedia rabbit hole. Definitely. Yeah. Yeah. All right. Well, Henry, thanks so much for hopping on this podcast and be sure to look out for him on the Patch Tuesday podcast. I'm Wayne and Miles. This is Hands on IT and we'll see you next time. Thank you.