Anxiety and depression - Bashar

[MUSIC] Welcome to Fragmented, an AI developer podcast that helps Vibicoters become software engineers. One episode at a time. I'm your host, Koushik. >> And I'm Yuri, the other host of Fragmented, where I'd love to talk about using AI to make you a better developer. >> Yuri, when we made the shift from talking about Android development to AI development, some folks asked me, are you still planning to keep the name fragmented? And my answer was resounding, yes, because the state of AI development is not really very different from Android development. It's super fragmented. If anything, the topic we're discussing today couldn't be a better example of why that name fits really well for AI development. The topic is keeping your agent instructions in sync. >> Yeah, absolutely. So many tools out there. There's so much things happening at the same time. One day I feel I need to jump into Cloud Code, the other day into Codex, the other day into Gemini CLI. Keeping all of these together is still such a big problem. >> It is. It is. And you still want to do it. >> But Yuri, what are we going to cover this episode? >> Right. So I think, firstly, we're going to talk about this sync problem. Essentially, how do we keep these things, these different instructions and configurations for your agents in sync? Like you were saying. Then we'll talk about solutions for providing this single source of truth. There's of course many solutions out there. And what we use personally. >> Oh, yeah. I'm actually super curious to find out what how you manage this. I'm always on the lookout to see how other people are managing this. So, can't wait to hear that. >> Right. And what is the benefit? And if there is any benefit at all in these instructions, right? Especially because recently there was this paper showing some mixed results. Yeah, so we can cover that. >> Yeah. Some people are calling into question if it's even worth having these agents instruction at all, right? Which is, right? >> Feel it's a little late to do that. >> Maybe let's find out. >> Controversial takes. Yeah. And finally, some tips on how to tame and control and manage these files. And our takeaways from all that. >> So, ready? >> Let's do it. Maybe one thing we should do is start, like you said, with the sync problem. What is the sync problem? Like, what is the issue we're trying to tackle today? >> So, the sync problem. Well, this was especially pretty bad in the early days, when all these harnesses started coming out there. But now, most tools have some kind of agreement. >> Yeah. And I don't know if people remember in the early days, like you're saying, right? There was agent.md, not plural, which is what we ever agreed on today. But agent.md, there was Claude.md, there was Gemini.md, so many of these files. And there were different ones too, right? I can't remember. Some of them had like a dot hidden directory where you could put in more instructions and it would pick it up. It was mess. >> Yeah. Everyone scrambling trying to figure out how to do the things. No standards. So, yeah, the industry is more or less starting to conversion that. So, now there's the agent.md, like you were saying. >> Yeah. And if people have listened to a previous episode, you'll remember that this agent's md standardization that we talked about is actually part of the open standard. The AIF foundation that we've talked about in a few episodes now. This was one of the early donations, which is agent.md, is the standard that everyone is pulling towards. And I think it was opening that eventually decided to rally people together. And then it was donated as part of the AIF foundation. But Yuri does one holdout. And unfortunately, they happen to be a pretty big holdout. >> Yeah. She was like the apple of the agents, right? Coming from 200 developers, we can say this. They like to do things slightly different. And to be fair, they have a very good harness and a very good model. So, I think that's why they can pull this much weight around. And they're definitely doing it. So, cloud.md is still out there. >> It's funny, Yuri. Just recently, there was a tweet where there's this popular web developer who, you know, publicly was a little, how do I put this? >> Dating. >> That's right. They openly called out and thropic saying, "Oh my god, I can't believe I'm not using cloud code anymore. I'm going to go and use codex. It's so much better." And they specifically called out how anthropic may not necessarily be the best development partner. And Tharik, one of these developers on cloud code, super nice person. And he's really puts out great tips about cloud code. He's active on social media. He responded saying, "Hey, what are some of the things that we could help improve?" And, you know, that's where the baiting comes in. I feel like he almost got baited because this other web developer straight up told him, "Why is this so hard? Why can't you consolidate around agents.md?" And obviously, you know, Tharik gave a non-response response. But he also mentioned, interestingly, he mentioned the technique, which, you know, I think a lot of us have followed, but I think we can take that as a good instruction on how to keep it, how to keep cloud and sync with the agent start. >> Nice. >> You come to that in a subsequent point. But one thing I wanted to call it also that's subtly different that many don't talk about here. Yuri is, there is a global level instructions, and then there is a project level instructions. A lot of this agent start md standardization that we talk about, usually referred to the project level standardization. I don't know about you, but there are certain times where I want a systems level instruction that applies based on my taste and my style. I want the agent to talk to me in a very specific way, and I don't necessarily always want to enforce that across the board. So anyone who works in the same repo as me, they may not want the same thing. Those I put in the global users instructions, and even that is subtly different from how it's managed. So, you know, there's a lot of these inconsistencies. Speaking of which, Yuri, maybe what we should do now is talk about how we do keep these things in sync, right? I'd love to hear your approach. What have you been doing to manage all of these different instructions and keep them in sync? Because I know for a fact you use open code, you use cloud code, you use codex, you use all of these tools, right? I don't think you're going there manually editing and making sure that these instructions are in sync in all places. So tell me how do you manage all of this? Yes, I do not, I do not do that. I do not keep copies, multiple copies of that. But what I like to do, like I think I was doing this, yeah, I've been doing this for some time now, the idea of having like a dot files repo. So this for people who are not initiated in this, what can I call this? This movement. It's a big thing online, you know, I think there's even a website. Yeah, absolutely, there's just people get people get, you know, very into these kind of things, you know, like these rabbit holes. I feel you're calling me out a little. If I'm calling you out, then I'm calling me out too. Because that's fair. That's fair. In any case, so the dot files, first I should clarify what the dot files is, but it's just the idea of having a repo where you keep all your configurations, right? Yeah, and this is not just for AI, this is in general, right? If you have a TMAX configuration, a terminal configuration, a ZSH, fish, whatever, all of those, you keep that in a dot files repo. Yeah, your A-list is everything like the things that the way you like to work on a computer, like you generally just carry this around. So the idea was like, I was doing that for my dot files in general, but I started doing it just for the agents and basically the configurations related to the harness. So not only agents now, the agents MD, but also like skills and all sorts of stuff. I'm trying to have this repo where I carry my agents configuration around. So this is the first idea. And now to keep them in sync, there are a few ways. I think we're going to talk about some. Well, what I've been just playing with and using is a tool called Rool Sync. It's an open-source project. I think we're going to leave them on the show notes. But the idea is that the guy built this tool that tries to manage all this confusion and like you basically can import and export configurations from each harness into another. So basically, you can import all the configurations from cloud and export it to cursor. So it will generate, it will just regenerate the configurations. And this takes care of both the global level instructions and the project level instructions. Yeah, you can run it on multiple scopes basically. Ah, that's pretty cool. And just to clarify on what you meant about. files. So do you keep a separate repo for. files just with all this agent stuff? Or is it in your existing. files repo? You have a separate folder now just for your agents. I actually created a repo just for the agent stuff. Which, yeah. So it's actually outside of my original. files. So I use this tool to keep things in sync. I think it kind of solves it for me. But there are other ways as well, right? So I think you wrote something about this some long time ago. Actually, I think before I even tried to fix this problem, I think you were already writing this blog post. What I did previously, I just manually manage it myself. Or we'll make sure to link the blog post in the show notes. But I also keep this blog post up to date because every time I make a change, I just add. So if you go to the blog post now, it'll show you the latest instructions. You don't have to worry about it being outdated. And I adopt a very similar approach to what you mentioned. So to cut a long story short, I have a shell script that basically sets up all the symbolic links and creates the necessary folders in the right way. So all I do is I maintain that. agents folder in my. files. And then when I need to set things up, I just run that shell script and it automatically just takes care of setting things up. So it's effectively the same as this tool that you're mentioning. But I'm actually curious about the tool because I would ideally not like to manually maintain this if someone else is doing it, you know, it's better to just use that. So I'm going to take a look at that as well. So people are probably listening to this and thinking to themselves like, why are you going to so much of this pain? You know, like is there any benefit? Is there any benefit at all in this? And I think recently this paper was published and it's a bit controversial like some of its conclusions. It was basically trying to evaluate the agent's MD. I think you shared this one, Kashi, can you tell what are the claims from this paper? In a gist, what these folks did is they tested specifically if these agents.md system instruction files are even useful right at a repository level. So people can go ahead and read it, but what we are going to do now in the interest of time is just summarize some of the claims and come back with like, you know, our thoughts on it. So the first claim that they say is auto-generated agents.md files make the agents actually worse, not better. They actually make the results worse. And here's the double one here. It also costs more in terms of tokens. Oh my god. It consumes 20% more tokens when you have agents.md files that are generated automatically by agents. So lower success and cost more. Take it up. The very first point is not encouraging. So that's one. We'll come back to it in a second. The second because you might have a good follow up, which is like, well, okay, what if I wrote it, what if a human wrote it and not an agent. What they found is human written files help marginally. And they say it's a 4% average. Honestly, 4% if you think about it is a lot. This paper doesn't claim it to be that big. So, but they say there's a 4% average improvement, but not consistently. So, for example, when they used cloud, the sonnet 4.5 model specifically, it actually performed worse even with human instructions. Again, this complicates things because which model did you specifically use and what were the instructions? So take everything with a grain of salt, but in general, they ran this study and found that human written files in general are better, but not always. So that was the second claim. The third is in the early days, people were like, my model is not even following this instructions. I have these agents start MD, but nobody's listening. What these folks who published the paper found out is that the agents actually do follow the instructions. The problem is like when you have unnecessary instructions, that's when it adds, they call it cognitive overhead, which I think is interesting because they're not talking about human cognition. They're talking about the AI's cognitive overhead here. The way they pointed out is the reasoning tokens take up to again 10 to 22%. That makes sense. So if you think about what is the cognitive overhead for an agent, the way they quantify that is a 10 to 22% consumption in reasoning tokens. And that can happen really easily if you have a global agents MD and you have a contradict. The local agents MD, right? So there are contradictions between both and the model gets even more confused. You pinned that exact point. This is generally the issue that lands up with all of these agents start MD files, right? Because the other thing we didn't talk about earlier on is, you know, the official standard for agents start MD allows you to nest agents start MD in every part of your folder. And the agent harnesses usually do a good job of like tree traversing through all of that and picking up all of those agents start MD. So very quickly, you're right, you can blow up that thing pretty high if you just start at a certain folder in your hierarchy. The second last one was context files are like largely redundant with existing docs. So what this means is these agent.md files when you have agent start MD and you had developer documentation like your general read me files and other documentation around the project that you have written for the human. You know, in the past when we used to write these documentation as good developers and engineers ourselves, basically they're saying it's completely redundant. But there's mild follow-ups here because say you started with a repository that didn't have any development like, you know, you were not a good developer bad developer. No, you know, no documentation. In those cases, the system instructions does 2.7% better. Oh, so it really does throw into question, well, so what if I didn't have the context files and I just had regular developer read documentation? Well, in that case, you know, it's a question to be considered. Interesting. Does that make sense so far like these claims? It's tricky, but hopefully the paper's claims are making sense so far. Yeah, very, very interesting to hear those actually. Eventually what the paper recommends is hey, just keep your agents start MD files very lean and just have the things that are different. If things are inferrable, like, you know, there's style guides, it's all inferred. But then again, you know, you pointed me to an open AI blog post recently around harness engineering where they talk about, no, maybe just give the overview, but don't go into all of the details. So again, we are here giving you the information that we see, you know, and talking about our experience, but don't treat anything and treat it as gospel, because, you know, everyone is trying to figure this out as we go along. Yeah, but this final recommendation, I think, sounds very reasonable. What I've seen also people do is to have like a huge agents MD and/or having like that, like you were mentioned, like this nesting of agents MD with contradiction between them. And it's actually really hard to keep these things up to date as well as the project evolves. Exactly. So you might have an instruction on one that says that this part of the code works like this, but then that's not true anymore. So the longer this documentation drifts, it impacts performance. So I think just the general recommendation, I have minimal stuff in there. I think it makes a lot of sense to be honest. I think that's the crux of it, which is keep it lean and keep it relevant. Don't add in-guide instructions, because then don't expect anything different. And so those with a general claims, people say, hey, you should still take these findings with a grain of salt, because there are some things about this test that we should talk about. A lot of the things that we care about and we put into our agents.md files are not always measured by some of these benchmarks. They have the sweet bench, these are common tests that people use to check models work. They don't capture some of these things that we ourselves put like stylistic preferences, those kind of things. So that's one piece that the paper doesn't necessarily cover. The other two big points that were brought up about the paper, one is they ran these tests on Python. And a lot of these frameworks like Django, Psychit, Flask, as we know, those are the best languages for these models, because these models are like trained the hell out of those specific frameworks and languages. So it would follow that in those cases, the models don't need as much help through the agents instruction file. Yeah, that makes a lot of sense. They already know. Exactly. But you and me are like Android developers. And as we know in the early days, a lot of these models don't follow Kotlin or Swift and these specific languages. So I would bet if you had more of those instructions in your agents.md file, like it would work better for us compared to someone who's working on say a Python project. But that's one caveat about how you should read through these findings from the paper. And last one is honestly the data set. They did it with the 138 instances and 12 repos. So that I mean, it's very commendable. But it's not what you would call statistically significant. I mean, you look at data. So that is like my summary from all of this, Yuri. Yeah, that's that's very very good to know like this kind of pushback and to see like both sides, right? Given all of this, Yuri, maybe what we should do is, you know, wind this episode down by talking about what did we learn from this? I think both of us still lean more on like we should probably do this. But can you help listeners sort of consolidate and put all of this information together? What should I start doing now? Yeah, I think I think a very good starting point is not to just run this last unit command. So do not auto generate and just walk away. Because like we were saying, there the documentation drift, you know, like you generated some instructions in the in the beginning. Especially if you use the command, it's like just auto generated stuff. So it's also not actually what you want to be doing. I think you actually want to have like your instructions there, like your vision of what that should be about. And I just have the model generate some statistical average of what what should be in there. Yeah, because it follows, right? If the model is already generating that, it kind of already understands that. So what's the point of giving the agent start MD the same thing? Yeah, I will say it's okay to maybe generate the in it, but that's more for the human to get a basic sense and then scaffold throw it away and yeah, exactly scaffolding. The biggest takeaway from that is what we were just talking about, which is keep it in lean. Do not over engineer. Do not over engineer the agents. Exactly. You know, you were just saying that I was thinking something. Is it the idea to have like the agent's MD become something like the constitution of the code base. So like very broad, like high level rules, you know, that's just guidance. I don't know. I love this idea. Maybe we can coin this term here, which is like your agent start MD should be the constitution of your project that the agents can follow. That is what they that's the law that they need to adhere to. If you think of it, that would have made sense. Interesting. All right, Yuri, so I think we've covered ground pretty well on, you know, the paper relevance, how to keep some of these tools in sync. Any closing thoughts? Okay, so I think my closing thought would be to not limit yourself on the tools, because in the end, we're engineers, right? And we've been hearing about the old tale of being locked into a database. I've heard that you have to abstract away the implementation details. I think we're in this very fast moving environment. And I know that a lot of things are popping up every day and it's really hard to keep up. But once you kind of understand what's going on, they're not that different. There is some there is some work you have to put into to be able to move from one to another. But they're not that different from each other. And in the end, there are tools that I think in this time and age, we should be able to to master. And it's like very important to have an hour or two belt right now. So I would really encourage you to first to try and to learn with them. And not to limit yourself, because you also give you you also get some freedom from you know, to be able to move because you know like the prices and how these models are and how these companies are. And I think it just gives you a lot of power to be able to jump between them. Very well said. Thank you all for listening. And we will catch you in the next episode. See you in the next one.