StaffEng with Karla Burnett: You're all 100x engineers...now what?

We talk about how the new frontier models and their ability to write code for us and stuff like that I don't know make sure like a 100 times engineer or 10 times engineer. I don't know like some extra order of magnitude maybe But it also makes your organization have all of the problems of an organization that's that much larger as well Right, so all of the security problems all of the like Order trail problems like you move that much faster and you get that many more things that you have to deal with Hey everyone welcome back to the staff end podcast. I'm Alex. I'm here with David Hey everybody And we have another guest here Carla Carla. Would you like to introduce yourself? Hey, yeah, my name's Carla. I live in Sydney. I have young kids I walk at a company called Larkey Which does customer service automation? So when you ride into a company the automatic responses that you might get back from an AI But trying to make them actually useful rather than just summaries of what you could already find on the website Um Awesome Um, so one of the questions we love to just start off with is uh You know thinking back like do you remember any early experiences you were having with LMS or AIs that really sort of like Opened your eyes to the to the possibilities Yeah, I think for a couple of moments that stand out Probably one of the earlier ones obviously like working at an AI company. I joined maybe two years ago So it was like starting to see like oh this could actually be used for something useful I'd seen you know all the like type of head sort of style stuff when I was uh working at my previous company when I was at stripe But I had me particularly impressed by it And then I think I started to see like oh this is actually capable of with the right harness with the right orchestration actually creating useful responses Then a little bit into my time there seeing you know Carsta launch and actually have quite useful type of head like multiple paragraphs of useful type of head actually Well, this is really saving me a lot of keystrokes Um, and then most recently Working with like the latest front-end models being like oh, I can actually do like full-on security analysis with this my background is I'm more in security engineering and seeing it like find vulnerabilities that I would never have had the time to investigate I was seeing it fix those automatically those I think have been some of the the more radical points for me I mean like the organizational Implications so part of the reason I'm really looking forward to this chat is because you know You mentioned it yourself working at an AI company Sort of probably been forced to grapple with some of this stuff earlier than Our peers in companies that where the product isn't sort of AI-centric And and so I suspect you're probably also a little bit further ahead in the curve of like operationalizing Some of this AI stuff internally. I'm curious sort of what that journey has looked like for Loric. He does a company and sort of Yeah, was the the impacts to your day-to-day in particular Yeah, I mean, I think early on it was just like early adoption of tools right like a lot of pushing of like oh like we've got Curse and let's everyone use cussers much as we can and then I think especially since maybe the start of this year It's been a lot more radical and that it's like okay like everyone really needs to be using these tools now This is no longer just like oh you can save yourself some keystrokes This is like this fundamentally changes the way we work and I think especially in like Maybe January a little bit late December a little bit early February those actually like a lot of eggs surrounded internally right people feeling like You know, do I have a job anymore right like if I write code but now the ais can write code for me What does that mean? And so I think there was a lot of like Both from leadership and maybe from more senior engineers like Handholding explaining like this is how this changes our job But you know, we still have an important role to play um and sort of Working to help make sure that everyone was on the same page in terms of the the tools that exist and the practices that you should be using Think one thing that we did particularly well is and this is probably because we're at AI focused company I'll see you like appointed someone as like the velocity's are right So like the person whose job it is to like learn about the latest AI trends figure out like test them Maybe we want to other people see if they're good and then like try to push those Appractices out to the entire rest of the company so that everyone because it's pretty healthy Because it's obviously like a full-time job at this point just to keep up I mean maybe not even for one person like there's so much happening all the time that I think one of the Really tricky things about getting any work done these days is like finding that balance between not falling behind but also like Doing doing the non-research thing that you're like supposed to be doing Right, and so it's nice to have have sort of someone who's who's delegated to that What's the process for sort of like you know you you mentioned sort of as an example shifting from like people can use cursor to like People really ought to be using cursor What's what's the process for sort of like I guess promoting that shift internally Yeah, I mean we have like a slack channel where we Sort of announced things that we would recommend. I'd actually say for example like these days we don't use cursor That's like we mostly don't write code by hand right um, so these days it would be more of a debate about like which particular flavor of codex or Claude code orchestration you like to use right would you rather use conductor or superset or just use it natively And there's a lot of discussion in a slack channel. We have dedicated to this about like what is the exact pattern that you put you in particular like to use I think it's just like something that comes up a lot. It's like very much a topic that's top of mind for a lot of people And luckily we are a small company. So we do tend to push changes out I think probably more socially than be any sort of like formal process other than saying like there's a slack channel follow this slack channel for like the latest and greatest hits I guess I guess also to be to be fair we also have things like we update like Quot empty or agents of D and like skills files like other things like that that technically do share things across the repo right and they'll like tag people We like hey, we've updated the guidance for example like to be like write more factually and less persuasively and technical design documents things like that So to a certain extent it's social and to a certain extent. I guess we push it out technically as well There's so many things in there um I so you mentioned conductor and superset Could you tell us a little bit more about those tools and like how people are using them in their day-to-day work? Yeah, for sure So for me personally I use conductor what it offers over say like raw code or codex is essentially for me at least two things one It makes it much easier to manage work trees So sort of it's got a side panel you can have multiple different work trees on the same get repo open at once doing different things You can have multiple sessions per each work tree That for me is very powerful because it allows me to work on a lot of features in parallel right? So I'll like kick off some very long running task like how I want to audit the entire code base for this type of vulnerability In another workflow or work tree I might say like okay, like go and Look at this linear project find all the tasks that are in this particular category go and create a PR for each one of them that addresses the issue that's there Right tests etc And so I find that that Is very difficult to do for me personally in lots of different tabs of code because I forget like which one is which and like where are they up to? Which ones need input and any given point in time And so I personally find the sort of like on the sidebar orchestration that conductor offers very helpful because it's sort of like You've got a list of what each work tree is doing and then an indicator of whether or not it needs your attention at that point in time Superset is a very similar product the primary difference between the two I think is that superset assumes that you want a terminal for it interface still whereas conductor is more of like I just want to check to something and so it might be more like the web app experience that you would have with Claude or Chechy PT for me personally. I was never a huge VIM user like I can use VIM it wasn't my IDE of twist And so I personally prefer the like chat style interface rather than a terminal, but that's very much a me thing And I wouldn't say I love conductor like actually there are many things about it that I dislike it's quite slow sometimes For example, but I think that the benefits it offers over the drawbacks it has make it worthwhile for me How have you felt about dealing with multiple long running processes I'm hearing more and more of people who are sort of like dealing with like the the load of now Of you know jumping from project to project like is that something you're feeling and is there any sort of like techniques that you're using to manage that overload at all Yeah, I think definitely trying to Try not to work on things that involve like lots and lots of context switching is something that I found helpful So it's okay to have lots of things on the go But maybe they're all related to one particular concept so like for me if I'm doing a security order to be like okay in this tab I'm doing an order of these things and then all of these tabs will like us work trees All of these work trees are working on like fixes for this other style of problem, right? I do definitely find these days. It's like okay What I actually spent a lot of time trying to think hard about is like how to deploy code safely, right? How to test incorrectness is maybe a little bit easier with security stuff sometimes because it's like if the If you're existing tests still work things probably are okay-ish, right? But how do I roll this out safely and that like they're gonna be okay-ish tests will probably pass But that doesn't mean that I can just turn it to 100% and assume everything is fine, right? So spend a lot of time trying to remember like which feature fires are in which state and like what exactly do I need to monitor for this feature Five to know that I haven't broken everything as a result I'm especially challenging because obviously we have a very non-deterministic product, right? So it's like oh if you just Change your prompt to make it more secure But then now you have lower quality responses verifying that you have changed the quality of responses can be quite challenging And so I've like started having basically like a little document that I have opened on the side of like here All of the feature flags and here is like the key log wine to look for and here is the current rollout state And then I sort of like keep that up to date at the same time I've actually put it in a notion doc now because that way someone else can help me roll it out And I'm not at work five days a week and so having someone else who can bump a flag for me and know what to look for is quite useful But I actually wish I had a little bit more process around like how do I manage that like deployment step a little bit better And that's actually something I would say it's one of conductors weaknesses as soon as you merge the PR and move the chat to a done category And I'm like it's not done. It's just much. That's a very different thing. The sort of like non deterministic Product I think is also pretty rapidly morphing into like You know effectively non-deterministic CI processes or non-deterministic automated review tooling and stuff like that. What have you all learned about managing non-deterministic pipelines that you think might be applicable for folks who are trying to figure out how to do this inside of companies that haven't had to grapple with sort of non-deterministic products, area previously? Yeah, we have a couple of strategies. So one thing we have for like just basic testing, like fundamental, like catch if you break absolutely everything is we have eVal tests. Traditionally, you wouldn't try to have like tests that hit a third party in your like test suite, but we found that we would break out like most basic sales demos so often that we were like, okay, we just need to have like very basic tests. They're like call the sales, like the flow that we would expect for like a demo account basically. And they check that like, you know, it's trying to make an appointment. Does it get back a date that was available? Does it get back a response that says like, yes, you can book the appointment essentially? Because even that, it turns out as like quite easy to break. Like prompts are very fragile, I would say. And so we added that and we run those on merges to main, right? And so we pay a cost in terms of time because we're running a new test suite. It's an expense as well, right? I guess it's not a free API at all, but certainly in terms of like minimizing breakages is helpful. Obviously that only catches the most egregious errors, right? Of like you completely failed to check that this didn't break a very simple flow for our actual customers. We offer, we have as part of our products simulations, right? Where we basically have like, okay, like take on this persona and then you've got like the LLM's talking to the LLM's, right? So it's like, if you like have a process that you want to test and it's on deterministic, the best way to test it is to add more non-determinism, right? But essentially it allows you to have sort of like a more human flow. And they are very useful, right? Because it allows you to say things like I've actually been doing some development of security stuff in the product and it allows you to do things like tests like, okay, like I added this guardrail for say prompt injection, right? Like go through and hold a normal conversation and check like have an LLM evaluate. Like did the conversation proceed as a user would expect, right? And it sort of allows you to take a step back and rather than trying to add a lot of determinism onto a process that will never be deterministic, instead be like, okay, what are the actual goals? What is the thing I'm trying to evaluate? Maybe I just have a front-end model, like something with the distant amount of intelligence, right? Know what my goals are and then evaluate whether that process achieved those goals, essentially. I think that's the best that we've come up with. But definitely, if other people have great advice, I would love to hear it. I think it's a developing space. Have you ever read anything about the company called Strong DM? No, I haven't out of them. So it sounds like they've done something similar where they, like it sounds like they build a fake harness is like slack, like a fake slack. And they have agents speak to each other inside of a fake slack and it represents their customer base. Anyways, it's just what if you haven't heard about it, like, you know, there's like what I love is it's like a it's convergent evolution, right? You both sort of like showed up at the same space, which I think shows that like it seems like there's a pattern developing here of like basically it's like just to make sure like I understand what you're doing, you're like giving the LM almost like a persona of your user or a user of your user and your user, and you're saying take, go use the product, try to, you know, make sure that you felt like you had a good transaction with us or something like that. Is that something, is that what you're doing? Yeah, yeah. So we would have like a workflow to find which might be like, oh, here is the, here's how you book a flight or here's how you like make a reservation. And then you'd say like, here's a user and they wish to book a flight or make a reservation on this date. And then you can mock out the backing tools and be like, no, that that reservation is not available. And maybe you, the person, I said they could also do this other day if they have to. And then check that they are able to as you say, get to an actual like walkable solution for that customer when they actually booked maybe not on their preferred day but on a day that was available. That's amazing. I think it's really interesting way to sort of like approach the problem. I'm wondering like, have you done any investigations into sort of like when a thing like that doesn't work? You know, like have you found a way to sort of push the quality of that type of tool at all? Yeah, I mean, I think it's still something we're working on pretty, pretty intensely. In terms of pushing the quality, we have sort of like an agent style harness within our product, within our product. So almost like talking to Claude code, but instead it's now like able to configure our product. And honestly, working iteratively with that is pretty useful. Like you basically like you can run these simulations, then you're like, oh, but this one ticket didn't do what I want, right? You go into this product we have called coach. So like, hey, help me diagnose why this ticket did the wrong thing. And it'll go through me like, okay, this happened because this thing was wrong. You're like, great, how do I fix it? Kind of thing. And we're like, oh, you could update like the guard rails for this or you could update the workflow like it will like suggest which everything makes sense. Make the update for you, rerun simulations added new simulation for that. And I think that sort of iterative process is probably the best thing that we've got for that right now. But I think that sort of having the LLM write the LLM prompts for the LLM sort of like, if you just add more layers on, you get some pretty impressive insights as a result of it. So in that world, imagine a ticket is like, represents an interaction with a customer. And so you have a whole this whole history, I imagine, between your agent and the customer. When you have it sort of explain what it did or try to assess, are you also giving it sort of like, what you asked some sort of LLM on the back end and the response you got back from that LLM so that it can sort of diagnose with the full picture in mind. Yeah, it doesn't get all of the details, right? Like obviously we've got system logs that go into like specifically exactly what prompted recent. I don't believe it gets that information. It does get like, hey, this was the workflow that you were running, right? So it gets us missing the like stuff we write out around the outside to give the context of like maybe the our users who are subscribed, I might have a workflow configured for making appointment. It's missing the like prefix and suffix style stuff that we might add to that workflow to explain like, hey, you're acting in this context. But it has all of the sort of like variables that go into that. So it's got like, here's the brand guidelines, here's the context for what this business is, here's the workflow that it's running. It's got all the events. So it sees like every tool that's called, it sees the guard rails that might trigger. So we have like steering guard rails. So maybe when it generates a response, it's like actually that was too uncertain sounding. So it will go back to the LLM and add extra things being like, make sure that you don't sound uncertain, right? And then come back. And so it can see all of those events and see, you know, whether or not a guard rail is incorrectly triggering. One I've been dealing with is like, yeah, this uncertain one is kind of like, where's the line of uncertainty if something goes wrong? Because it's like, I think something might have broken. Is that too uncertain for a support agent? Is an interesting question that's very business specific? So sort of customizing that and saying like, no, you said this was too uncertain. Sunson, I disagree, right? Like make it so that this guard rail is less sensitive. And it's able to iterate on that and use simulations, right? Like it'll add those cases for you automatically run them. Generally, it'll iterate that way. You've mentioned security sort of tangentially. I mean, it's clearly an area of focus for you specifically and we've been kind of poking around it at other things. But I do think that the implications of introducing LLMs into the workplace, obviously come with unique security constraints. I also think that it's possible that right now, given the, I think, FOMO that a lot of tech leaders are facing around LLMs, it's maybe prompting a little bit more YOLO behavior than was previously sort of acceptable. And I mean, I guess what do you think are the scariest things that organizations are liable to do with LLMs usage internally? I mean, I guess also more broadly sort of where do LLMs products get scary from a security standpoint? But given that a lot of the listeners are going to be integrating specifically coding agents and also sort of like internal productivity agents, where does that sort of, what does that line up for you? Yeah. I think it kind of cuts both ways. So I've seen companies that are very AI shy because they worried about the security implications. So like, no, we don't use any AI products. And I think that will be to their detriment in the long run, right? Like I think you're losing out on so many benefits. Yes, they're obviously trade-offs and you need to mitigate risks. But I think that's a silly way to approach that personally. I would say the biggest risks right now on how many systems we connect with AI agents that we haven't previously thought about, right? So before I might be like, oh, like I have a Slack integration over here and I have an integration to like, I don't know, community secret manager over there, right? But like, never the two, she'll talk to one another. Because why would you connect Slack to the GKE's secret manager? The answer is now like, well, that both MCP's have is that an engineer might reasonably need access to, right? So how do you stop the secrets from ending up in Slack by accident, right? How do you actually like put appropriate guardrails in place? And I think like, Claude code and codex attempt to solve that by like, prompting users a lot, but they prompt them so much that people just say yes all the time, right? Well, they use, for example, conductor, which has dash, dash dangerously allow all and just turns off all the permission checks so it could kind of just do whatever, right? So think like the overall security model for MCP servers is probably what scares me the most just because you put this thing in the middle and like even like assumed that the agent is doing with like, it's trying to act in good faith, right? Which may or may not be the case, but let's just say that it is even just given that it's acting in good faith and misconstrues things that you've said. The fact that it connects like all of these very sensitive things with all of these places that are very publicly accessible and also in a fairly opaque way, I think is some like a risk that people sort of overlook slash like willing to just turn a blind eye to right now because they're so powerful, but that's what concerns with the most true and organizational standards they say. Are there any patterns that you've seen that are like useful in mitigating that risk? I mean, I think always the best pattern is to assume that the LLM is like trying to be nefarious even if it is not, right? Like, you know, the just because Occam's razor says that it is probably not trying to be nefarious does not mean but it doesn't in fact accidentally do things it shouldn't. I've had LLM's like something. - I already hear her first voice. We've reached takeoff. AI is sticking over. - I mean, I've had LLM's merch stuff to main without explicit consent before, even though they're told not to, right? Because they're like, "Oh, well, I thought this thing "you said earlier meant that you wanted me to merge domain." But it's not what I meant, and now the code's in prod. And it's fine in this case, but it's a little concerning that you weren't bothered by this, right? So like, obviously the best thing to do from a pure security perspective is like limit the functionality that it has access to. So, you know, for example, with secrets, maybe say like, "No, you can't get access to the secrets. "You want to give it access to the production database "so that it can do what it would like diagnostics "on all of your customers and things like that." Also, fine, but maybe encrypt any secrets that you have to store in the database so that like, you know, additionally encrypt them so that, "Hey, I wouldn't have access to that," right? Essentially like the classic security thing of like, if you don't have access to it, you can't steal it. I think that's probably the best you've got there. You can add hints to MCP servers, indicating that they're destructive or read-only actions, but they aren't really hints. And so they're only as good as, you know, LLM checks. I think anthropic and open air are probably working pretty hard on how to make permission checks better, but it's a fairly unsolved problem. I would say within the like, entire human factors elements of like computing, how to prompt appropriately for permissions, the ideal thing obviously is to prompt when something needs it, which is what Cloud Code does do, right? Like it's similar to the like, you know, mobile phone permissions interface, but there's so many things that it has access to and knowing what is or isn't sensitive can be quite a challenge, especially if the person is asking the questions to isn't themselves technical and doesn't themselves understand the security model of their own machine. So like my ideal thing would be if we could all run, do our development on a, you know, entirely isolated machine where you could just give it dangerously do whatever it wants, right? And only mounted to that container, whatever you need. But I've yet to see anyone doing that in effective way. I think like even, you know, there's certainly folks who have, who are dev box style stuff where you have a machine in the cloud and the agent is running on that machine, but I think your point might be that like, it's difficult in most cases to do productive development work if you don't have access to a lot of these tools. I also think there's a certain irony where, you know, I'm kind of in a very early stage startup environment these days and the sort of default move in the very early stage startup is like, you know, very high trust environment give everyone access to everything. And that works great when you have a lot of really sort of conscientious humans are working together. But I'm finding that I'm actually reaching for tools that have sort of like enterprise-ish granularity on permissions so that we can constrain the agent's access, right? Like yes, give the human's access to everything, but then like I want very fine green permissions on the coding agents and it's sort of like a funny realization because it, you know, prematurely increases our bills with a lot of vendors. - Yeah, I mean, I think like we talk about how the new front-end models and their ability to write code for us and stuff like that, I don't know, makes you like a 100 times engineer or 10 times engineer, I don't know, like some extra order of magnitude maybe. But it also makes your organization have all of the problems of an organization that's that much larger as well, right? So all of the security problems, all of the like audit trail problems, like you move that much faster and you get that many more things that you have to deal with. We've actually had like fairly good results, I would say, with adding like security paved roads to our agents advice. So for me, especially not working full time, I'm the only like true security person at my company, it's still small enough that there's just me. I'm being able to put in paved roads that it's like when someone tries to do something that they don't realize is unsafe, there's some guidance where the agent is like, "Hey, like you're trying to ship data to a new third party "that you shouldn't be like, did you mean to do that "if so, it's fine, right?" But just like have it call it out. And I've actually had it do it for me. It was like, "Oh, hey, you're like shipping extra analytics "dotted to logrocket here and you're not supposed to do that." I was like, "Huh, good call, this is a demo app "so it makes no difference." But it was good of you to point that out. And that has been helpful at the very least. - Could you use that? - Yeah, so go ahead. - Could you say a little bit more there? When you say paved road, what do you mean? And like what does that look like to tell an LLM about a paved road? - Yeah, so an example might be, let's say that someone who isn't an engineer might background, right? Like maybe they're like a Ford Deployed Engineer or maybe they're like on the sales team. Now they've got a code with them, they want to build some kind of demo thing, right? So the classic for us would be when we do sales prospecting these days, I think everyone gets a custom demo app, basically, because like why not? Right, it costs nothing. And so obviously that now means that there's a lot of quite non-technical folks who are technically writing code, right? Like they are going off and shipping these things. And the natural next step for those people is like, "Oh, I want a dashboard," for example, that shows like how my sales stuff is going. I want a dashboard that shows this one custom thing for this one particular customer, and that's a very reasonable thing to want. And then the question is like, where do I deploy it? How do I do access control for it? How do I make it so that this works for this customer? And obviously the easy thing is like put it on the public internet. It'll work great. People usually aren't so naive as to do that, but they might say like, "Oh, I'll use this like platform as a server style company that just grows off and lets me host code really quickly," right? And that's fine if it doesn't have any sensitive customer data. But if it does, now you've suddenly got this new vendor that's hosting customer data that you shouldn't have been using and you had better ways of doing it, right? So that you hear is rather than leaving everyone to the wolves to figure out the best solution, you've documented your agents.md file like, "Hey, like if you want to host data of this shape, "and like this is the problem you're describing here, "is the way that we would recommend doing it." Basically here is a template project that's been set up. Here's like, we tried to optimize it for like, in our case it's like, okay, maybe it's okay to require an engineer to be involved the first time it deploys, but not every subsequent deploy, right? How do you make it so that you sort of balance that trade off of like, you just want to check that someone has slightly checked the configuration of this? We thought it's like letting people move fast. And the same thing goes for like, oh, you know, we have real-level security in our database rate, so one particular customer can't access other customers data, right? You can bypass that if you try it, it's got obviously like ways of turning it off for particular tables, which might need to have cross-customer data for like, I don't know, maybe like the customer table itself, obviously, is like we can't check that you're authenticated. So if a customer until we've checked who you are, that has that security turned off, and then we expect extra rigor on those queries. Telling the agents like, hey, don't turn that off just because it solves your problems the rest of the time, right? Because my general model for LM's is like, they will try to help you solve your problem, and they weren't necessarily thinking about the externalities of trying to help you solve your problem, as quickly as possible, right? And so it's like, yes, it's true, if you turn off authentication on this route, you will find that you can develop it much more quickly, right? But ideally, like, everything should be off - It's not all that different for human engineers. It's really very similar. It's like my model for this. - The security stuff is quite pesky. - Yeah, I have a very like junior, like a very enthusiastic junior or maybe mid-level engineer, and they are absolutely convinced that they want to ship this thing today, and they cannot question any of the assumptions that they have been given, because they were given by a senior engineer and like, you know, having forbid that they choose to question any of those things, right? What would I put in place such that I wasn't scared about what that person could do, is what we try to encode in like agents.MD files, right? So it's like, whatever you might have was like best practices, I guess, equivalent documents. Well, I could get let's stick that in agents.MD file, have an LLM edit it for like, you've got an LLM audience, right? So break it up appropriately, like, try not to shove too much into the context to start with, just what you need to know for them, where you should look for like security guidance, or for example, prompt engineering lives in the directory that has all the prompts in it, right? So it only needs to load that in when it's actually doing something relevant to that area. - That's cool. So when people are trying to create these like demo apps, how do you make sure that they're like getting the shared agents or cloud.MD files, especially if they're deploying someplace other than your like, where production would go? How do you make sure that people are getting all of the LLM knowledge? - Yeah, so for us people are usually using Cloud Code in one of two repos, either a personal productivity repo, which has all of the like skills that are not specific to our sort of engineering practices, and then also our like main engineering monorepo. So for us, we can like, we just put this guidance in one of those two places, and then people will get it basically by default. It also just happens that like when people are shipping customer data to places, they would need to have access to like the databases, and that's granted in the engineering monorepo config. And so like, we've not had a case for anyone's been off like shipping customer data by hand somewhere else, 'cause that would be slow and time consuming. And they're not trying to bypass these processes, they just don't know that they exist, right? So if you sort of put it in the same place that you get access to the data, then it just comes bundled for free. - That's interesting. So the personal space is that does everyone have their own or are you sharing this sort of like this fuzzy space that's not like engineering production, but it's like this like sandbox or test bed for sort of LLM stuff. - So this is like a personal productivity repo. So it's got like skills that people might find useful across the company, but that are not specific to any given one person and are not engineering related. So an example might be like, oh, like post an update on how my project is going, and it will like ask you the relevant questions for like whatever format we're using that week. In terms of like how the project is going, your project updates to Slack, ship them to wherever else. We want them to go similar calendar management style tools. They all go in there. There are in both repositories like a dot playground or something directory that's ignored so you can stick extra skills and things like that in a place where they're just for you in particular, but we're very encouraged to like push the changes out to everyone. So we have like a skill, for example, which is like ship things. Specifically that creates a PR checks for automated PR feedback, checks for the CI monitor fixes all of those things, addresses any PR stuff, specifically asks to merge, merges then does like post deploy monitoring for you. All of that sort of thing. Unless you're using conductor in which case it marks that is done and moves on. Well, it hopefully merges it for you and then does the post deploy monitoring and you're like, I guess it's good that you monitor the code that you accidentally merged. But yes. I was curious, you mentioned a couple of like fairly long range tasks. Are you using anything special or are you just using a normal ticket management tool to sort of like encourage agents to work more long range or work on a broad set of work? I don't think it's anything to do like ticket management stuff. I think a lot of it is sort of like how you. I guess that's not true. We use superpowers right to the Obro Superpowers plugin. We use that pretty heavily internally. I wouldn't say everyone's using it but I think almost all engineers would be. So that obviously encourages a particular style of development which makes it a lot easier for you to like essentially what it will do if you like give it a very vague task. Is it will ask you questions, create a design doc and take like you review the design doc to be like color write an implementation plan takes the design breaks up into a series of stages and then as like how would you like me to implement this and you like sub agent and then it just goes off and does its own thing for each of the different steps. Have you ever added sub agent have you heard not selected one on that question. Yes, I've selected the other one a couple of times. It's been incredibly painful. Don't do it. No, the other thing I found is like you use the whole superpowers flow and it's like really good generally very excellent. But after it's finished doing a very long range task, I usually then open another chat in the same work tree and I'm like look at the design doc look at the implementation that exists yet. If they match, are there any to do's because I've definitely had a couple of times of like long tasks where it's like and then like to do implement this hot thing and I'm like no, I want you to do the work. That's the point is you know hi agent is that you do it for me and so definitely like almost like if you've heard of like Ralph which is like the idea of like just giving the same prompt repeatedly. I feel like I do that a little bit manually after superpowers runs and just be like are you sure that everything is covered here. What's different between the two. Okay, those things are minor and I don't care about like rate limiting in particular on that end point. But like doing that I think is like been surprisingly effective as a fairly simple strategy to actually get it to properly finish tasks. I think like that works quite well for I want you to implement this big feature that I understand because it will go through and spec out features off doesn't work as well for like security audit style tasks. So in those cases I've usually found it better to do like go off I do a full security audit give me all of the things that you think could be issues. Okay, write them in an MD file now in a new chat go through and for each of those in sub agents or whatever I don't care. Generated proof of concept and check that that actually is a valid vulnerability. Okay, it is now go off and create a linear task for each one that is a valid vulnerability in this project. Right. Okay, now you know like a new entirely new work tree or new context at least like okay for every single one of the things they go off and create a sub agent that fixes the problem. I think like usually it's mostly about like context management. So like you can ask it to a very like broad task and the thing that you tend to find is just that at some point in time it loses context. So for the security audits for example I had to repeatedly be like are you sure that's everything. Think really hard are you sure that's everything in that category. And even then it was more effective when I rather than saying like find vulnerabilities was like okay find vulnerabilities of this shape just because you know it's kind of like a human and so this is only so much can remember it in a given point in time. And so asking it to find things that look like a specific shape tends to be more effective and works quite well with this sort of like it's for this of like writing all of the findings that you have into like a notion or linear notion, like notion, like linear project and then going back and subsequently going and fixing each of them. And I guess to your point of like how do you keep track of all the projects. It's a little bit easier for me personally if I'm like okay the thing I'm doing right now is like lots of deep auditing. Okay now I found all these terrifying things now we're going fix all of the different things as like different strategies to keep yourself in the one head space I guess. You mentioned earlier in passing something about a prompt for the tone to use when writing an internal document I think you said something about sort of be more fact oriented instead of persuasive or something like that. Yeah. And so the implication there is that many internal documents are drafted in partnership with with agents. How does how does that process work. In terms of like drafting documents in tonally and then sharing them when they're written by AI or. Yeah, maybe to add a little bit more color I think something that I've experienced and that I think others experiences that like AI agents are greater writing lots and that can lead to a dynamic where you have one person using an agents to generate a bunch of pros and then another person using an agent to summarize a bunch of pros and the agents are basically playing telephone between the people. And I think that there are also effective ways to use agents to write documents but I'm just curious like if any of what I just described resonates and you know sort of what you all have learned around that. Yeah, I would say that we aspire to and usually hit the guidance that you shouldn't be asking another human being to read something that you haven't at the very least edited right to the sort of like that equivalency of work right like you shouldn't ask someone to do more work than you put into a document right like reading it shouldn't be harder than writing it. And so I think what that means is like we try to keep the number of documents that we share amongst ourselves similar to what it would have been sort of pre AI but maybe you just save some time on the writing and get to spend a bit more of your job doing something else. But you still obviously need to edit it. I guess an example of like where I don't driven stuff has been super useful for me has been like I often like security audits had it right up a threat model for me. I don't like writing threat models I like the concept of threat models and I like using them but writing them up is stupidity is boring because it's just a catalog of things that could go wrong. So I sort of like drafted within lm a threat model that I don't really intend anyone else to read it sort of more for me to think about the problem. But then I was like okay given that we then implement these fixes for this threat model right if we had all this be true right me the sales pitch for marketing right me the sales pitch for like a customer. Keep it short right like I don't want it to be long it's something we would show a customer and then I went to a marketing team and I was like hey would you be happy with us saying this right because obviously like the marketing team wants to be like we are 100% secure our agent will never ever say anything it shouldn't and it's like okay you can never make that guarantee right like I could tell you that we will never like share cut data between customers but can I tell you that it will never like randomly spit things out that it shouldn't know of course not right like I can do my best and I can like put in these like product protections but I can't make that guarantee 100% and that's obviously like a way to do it. And that's obviously like a more subtle message that I wanted to make sure other less technical folks at the company was on board with before I put a lot of effort into like building defenses that they were not going to be happy with. And so for me it was very helpful in that case I guess kind of counted the thing I just said like basically I told a dry and it was way because it's sales pitch right a sales pitch and then like show the sell like the marketing team like does that sound like the sort of thing we're happy to say okay great now that we've got that I'll go off and like actually bother working on that because I know that we're all happy with the place that we're going to. Whereas I think the counter argument would be like if you have like an RCA for an incident right like opposed to one of her incident you don't want that to be written persuasively right like yeah it's totally fine to have the AI draft a bunch of it we often do right like timelines are not something that I need to manually curate to get the value out of right like if it's an AI is going to be way better at copying the exact time stamp that something went wrong and slack or it went off. I need to write like a brief recalls thing is useful but sort of going through in like editing it and make sure it has the right context for the business situation is something that it needs some human guidance on and hope human oversight often so we would make sure that like those parts were written by human before putting them in our like great archive of things that have gone wrong for later analysis probably by an AI. Are you doing code reviews of skills. I wouldn't say it in a particularly formalized way I don't think we have a great solution for how to do like skill testing and skill improvement other than just using them and seeing what goes wrong so that's by and large what we do we have automated code review from code X I think on every PR so technically that would be some amount of review but I wouldn't say we we go into any particular depth on it no. I am most curious because I think we're all starting to now use shared repositories of skills like you mentioned superpowers which is like one that I've seen really take off for a lot of different engineers right but then internal it to teens even you know so there's some hierarchy just being care I'm really curious to see how this is going to develop you know like I have a strong sense for how. Ruby should be written or Python should be written but like how a skill should be written it's like everyone is equal at that table you know what I mean it's going to be really interesting. Yeah one thing that we've seen is that it's more effective to describe what you want to an LLM and have it right the skill for you than it is for you to write the skill because it's a new area turns out that actually like the LLM themselves tend to be better at doing it the humans because we have no practice and they have at least some training data on that and so like that's what I've mostly been doing I'm like I don't write skills myself by hand because I get them wrong I have the LLM right them for me and then I go through an edit them if I see fit but even then I would usually do that in partnership with an LLM because it. It'll think of ways that the LLM gets things wrong that I might not then you know if I think of things I can easily tell at that and then it adjusts accordingly. Are there any other skills like superpowers that you see are being used broadly in general. I think superpowers is the most like is the only one that's like publicly available that I see being used broadly obviously we have like custom ones for our repos that are used quite commonly but they're very specific to our business context. I don't think I know of other ones that I see people using a ton specifically. Neither have I for what it's worth. Maybe maybe anthropic skill creator skill like to you to write skills you know what I mean but not it the superpowers when it's really interesting to me because it's coming from outside of the of the frontier. Yeah, it's sort of feels like it's like the first like jQuery of like skills is like the superpowers where you're just like me to use this thing like just use jQuery just put it on the page. I mean I think it's an interesting example of taking like methodology of problem solving and writing it down because that's essentially what it's doing right like it's just saying like when I solve a problem this is the way I do it. And so I think like you can come up with other examples of where it's like the issue isn't necessarily the knowledge it's like the patterns that you use for development. Well, the patterns that you use to achieve a task. And so I think you'll probably see, well, I would like to see the same thing a much like different areas as well, right? Like you can easily imagine like, here's how I do UX research, codified as a skill, or here's how I do-- It is interesting that there's-- --sticular things. It's basically, yeah, it's like, it's codified tribal knowledge, right? Things that would have previously been sort of like passed along probably verbally inside of teams and, you know, from sort of like generations to generation of engineer. I mean, I mean, I'm making it sound like there were many, many generations before us. But, you know what I mean? Now, it's like, it would have always maybe been a best practice to write some of those things down. But now there's sort of like an incentive and a mechanism by which not only will people write them down, but like there's sort of value in aligning on a few that everyone kind of agrees are good. So it's interesting to see that play out. One of the skills that I have found interesting is, Versel has a skill repo. It's at the skills.sh or something like that on website. But it comes with a find skills skill. Carlers terrified. It's a security nightmare. But it's interesting from the perspective of like, it does allow you to find things like this. Like, there is a UXR skill in there, right? There is a whole bunch of web design skills in there and stuff like that. And so from sort of just like, the perspective of being able to quickly pull these things off the shelf and play with them a little bit, I've used sort of like copywriting skills from there and it allows me to be like a passable copywriter for, you know, a marketing pages or whatever. So there's a kind of interesting interest. Yeah, I think superpowers as like a visual design skill that I use pretty intensively as well for like small demo things where it's like it will want your browser for you and be like, here is a mock up of like two or three different things. And for someone like me who like, I would not describe front end as my strong solution, especially not like good design. I'm like, yes, let me pick between three options to tell you which is wrong with that one option that I chose and what I want you to fix is like, that is the level of design that I'm capable of. I cannot like to know where come up with a good design for you, but I can vaguely tell you that that looks better than that, right? Or is more what I'm going for. And so I find that very useful, but I think it's still technically part of the superpowers, which maybe is just an indication of how much superpowers itself is also growing, right? Like it's way more than just the thing I described, I think it's sort of like the core loop. Yeah. We're about a time here. So we're going to ask you our last question, which we ask everyone, which is how much code are you writing nowadays? Like manually written code. I think it's up to you how you want to answer the question. So there is some history here. We used always the previous season of this show, which was four years ago. We would ask people this in the context of like, oh, you're a staff engineer. Do you actually sit at your keyboard and get to write code yourself or are you just talking about writing code? And I think nowadays, whenever we ask that question, people, you know, look at us and kind of like, do you mean like literally writing code manually, the old faculty? And so I think you can take it kind of however you want. But like, I think the intent behind the question is like, to what degree are you thinking about, um, about like the code layer versus all the other shit that goes into making software? Yeah. Um, so I don't know if small startups like code write more code. So I was writing code even before I was using like all the AI tools again, but I certainly write a lot more code these days with AI's help. Um, I kind of think of it as like managing a team almost of like people who are like going off and doing implementation. That's the closest analogy I found to like previous work. Um, in terms of like writing manual code, never without best tweaking of prompt that I found somewhere, but like essentially never would do more reading of code these days, right? Obviously for code review, but even just like when there's, I want to find the log line that's relevant. Maybe I know exactly what to search for. And I'm like constantly confounded by why like conducted does not allow me to globally find within my code base. I don't need to change anything of the files. Let me find things in the, like, repo. But yeah, I would like read code a lot more than I would ever write code manually these days. We had an interview the other day where someone's conductor or something had been working in the interview that we were giving them. And we were like, Oh, you wrote code like real bespoke code. And it's like, this is ridiculous. Six months ago, we would never have used the term bespoke for writing code by hand. Artisanal. Yeah, exactly. It was like, Oh, a seasonal code, right? But that's what it feels like these days. Person showing off writing, writing assembly lined by line in their interview. Well, thank you so much, Carla, for joining us. It was great meeting you and getting to learn from your experience. Yeah, it was great chatting. Thanks a bunch.