Inside the Dark Factory: AI That Ships Code Solo

Two weekends ago, we had quite a busy Friday, and the dark factory ended up shipping 150 PRs for two of us over the course of the weekend. And that was just what we were out hanging about, enjoying London in the heat wave. Last time I looked, I think about only 5% of the dark factory codebase had ever had a human look at the PR and to prove it. I don't think people actually go look at code anymore because they're all just using agents. And so that means by implication, 95% of our codebase has never had a human look at it. The AI Native Dev is a podcast for developers and engineering leads at the cutting edge of AI and agentic coding. Join your hosts, Guy Pajani and me, Simon Maple, every week as we chat with the most exciting voices in AI and tackle the biggest questions facing developers today. This is the AI Native Dev. We just wrapped up two amazing days at AI DevCon in London. But the great thing is that we get to do it all over again in New York City this November. You're absolutely right. We're going to be back in the city that never sleeps on November 3rd and 4th for more amazing sessions, really engaging hands-on workshops and much more. Yep, all that great networking, partying, eating and drinking that you've come to expect from AI DevCon. We think we have one of the best hallway tracks in the business and it's the perfect complement to our incredible speakers and presenters. We'll both be in-person and virtual with live streamed access to all mainstage keynotes and talks. Sign up right now for our super blind bird ticket for just $100, only available for a limited time. We're really excited to be headed back to the Big Apple. We hope to see you all there. Hello and welcome to another episode of the AI Native Dev. Did you know that last week alone Tessel raised 516 pull requests and the whole company was at a week long offsite. The week before we seemed to raise only 608 so only 90 or so more. How do we do this? It's all about the usage of software factories and the Dart factory. Joining me is Rob Willoughby who is the AI engineering lead at Tessel and Rob's going to tell us all about how Tessel uses Dart factories and software factories internally to create such amazing numbers. Rob, welcome how are you? Thank you. Thank you. Doing well. And first time on the AI Native Dev podcast. Yes, it is. First time on a podcast. Right. We have to do the induction unfortunately. So no, there's no induction. I was just thinking of making something up then. Yeah, few. Didn't tell you that. So Rob, first of all, tell us a little bit about what you do at Tessel. Yeah, so I run the AI engineering team here, which is part of the thing that kind of broadly is looking at agents enablements kind of across Tessel. And so what our mandate is is really to build and then drive adoption of a dark software factory internally that we use for kind of two main reasons. One is to improve our own engineering velocity. We want to ship faster, ship more, ship higher quality stuff, be able to use agents better. But then also additionally, we use that as a testing ground for our own these season hypotheses that had effectively run a factory. We want to be able to take the learnings from there and put those into our products that we can then go help other companies bring along on the adoption curve as they start to build these factories internally for themselves. So let's, these numbers are, they seem crazy, right? Every time I take towards people and say even a few hundred poor requests for, you know, not a huge team, not a small team, but, you know, a reasonable size team. People are surprised. They're like, how do you deal with that? How do we deal with things like, you know, code reviews and the human interaction and things that will cover a lot of this and actually during this podcast, we're going to talk about how Tessel adopted a dark factory, what were the sticking points, what worked, what didn't work, how we overcame them. But for now, let's actually take a look at what a good interaction looks like through the dark factory. Let's now go through a full end to end test of two poor requests, one which requires human interaction and the other which will go end to end without human interaction, starting from a linear ticket all the way through to a completed poor request. So this ticket is one that I prepped beforehand, which has to deal with some findings from an audit we did where we can accidentally allow data from GitHub repos and inject commands into the Tessel agent commands that we are showing on the website. That's just the city with string escaping and all that kind of stuff. So relatively simple fix but security important. This is going to be touching what shows on our website. So it will require a human PR review in order to land. All I've got to do is I've got this ticket prepped saying it's in apps front end. So that's our front end, obviously. And then I'm going to flip it into to do because it's been delegated to the dark factory. That means that the dark factory is going to pick it up and go do the implementation for me. So that's kicking off the dark factory. Just putting that into it to do the dark factory recognizes there's a hand over there and it kicks off a new container where it can kind of like do with that. Exactly. And so now I'm hands off at this point. So the second ticket is one that is targeting or dark factory itself, harness slash geekymora. That's just our own internal name for it. This is another one that is around expanding out our metrics. We aren't actually capturing which inference geo where we're running these queries against, which means that we're not able properly analyze the effects of caching because our core quest could go across any different geos, which then effects cache rates and also cost overall. So because this is within our own dark factory, we have a policy of auto merging everything because if we can't auto merge it, that means that we don't have the verification in place to have confidence in the process of writing code or building code. So you can see that this auto merge label is down here. This means that as soon as the dark factory is done creating it, we'll go through PR cycle process using bots, of course, no human intervention here. And then we'll get auto merged into the repo once all of CI passes and once our review agents prove it. So I'm going to flip that into to do and we'll check back in on this one at the end of the podcast. Amazing. And the fact that that is set to auto merge, if that wasn't there, presumably that then would require a human to go in, maybe review the PR and actually push it themselves. Yeah, exactly. So auto merge is an optional label that just gets the bot to literally select auto merge on GitHub PR so that it will go in once CI is green. Awesome. Amazing. So now we've raised two tickets, two linear tickets. One, which requires a human review. So the dark factory will do all the work and then it will say, this is done here's the pull request, prove it and it will be it will be merged. The second one, because it's a PR that goes directly to the dark factory. It's a PR for the dark factory that will auto merge so as soon as it's done, it will it will squash that down and and merge that into the code base. Yeah. Amazing. We'll come back to that at the end of the podcast. So what do we talk about how Tessel uses the dark factory numbers wise, first of all, and then we'll maybe look back at us to what we were doing before the dark factory existed. So when we think numbers wise, what percentage of Tessel's pull requests today use the dark factory? Yeah. So varies a little bit week over week. But for the last week, we are between 65 and 70%. Mind you, this is including pull requests that are building the dark factory using the dark factory. So there is a lot of those as you can expect. We are heavy users of it internally on our team. The last time that I looked for the production part of our code base, so the stuff that is serving the registry or web UI, our CLI, all of that kind of stuff, we are at about 40% of production PRs were raised using the dark factory. And all of those 40% require a human review to some extent. Correct. Because they do. Our policy is that we only allow auto merge on stuff that is on our research code base, stuff that is on our dark factory code base, and stuff that our GTM team is building using the dark factory to help build out their own internal flows too. And how about weekends? Does the dark factory work while we sleep? Oh yes. We'll happily work weekends and overnight as long as there is work in the queue. It feels like a little bit of the shift in my own thinking about this is less about what is the coding work that I think I can do over the course of the week, but what is the coding work that I can be able to effectively feed the queue over a long enough time? Because we respect linear blocking relationships and able to handle kind of subtests, and all that kind of stuff, I can actually sequence quite a fair amount of work before I go away for the weekend and come back to not a fundamentally different architecture, but a relatively different architecture over the course of the weekend. I think two weekends ago, we had quite a busy Friday, and the dark factory ended up shipping 150 PRs for two of us over the course of the weekend. And that was just what we were out hanging about, enjoying London in the heatwave and some of the sun here. Was there any interaction that you had over that weekend during those? No, so those were all auto merged in. Amazing. So Tessel didn't always use a dark factory, of course. What was the engineering styles and culture pre-building the dark factory? So the way it fermented was single-player agentic engineering. In terms of you had people have their local agents, or maybe a cloud agent like CloudCode on the web or cursor agent in the cloud or anything along those lines, but it was still ultimately being driven by an engineer and the interactive human or the interactive driver seat. And so you'd figure out with your team what the work you were on to do, you'd then work with an agent to scope that work, do the implementation, test and verify locally, push up and merge the PR. So very similar to kind of the previous flow, prior to agents, prior to LLMs, just on an accelerated time scale because the implementation was now getting cheap. But it was still that very much so I'm a human driving a ticket through to completion with many touch points along the way. Yeah, so it wasn't so much workflow change. It was more like you say, it's an acceleration of that code generation really. And I guess what were the tripping points at that stage? What were the things that would slow us down? Because I guess speeding us up with the same workflow is going to cause problems. It's going to cause a different bottleneck in another place. What were the problems, though? Yeah. So I think verification and testing was a big problem. So we run a Mono repo where we have a bunch of build dependencies. And so in order to test it effectively, test everything, you've got to build it locally. And then you need to walk through a whole bunch of-- go hit a bunch of endpoints, or go click a bunch of buttons in the UI, or something along those lines. And so that meant that any change that you were doing-- yes, now your implementation is free, or functionally free. But you still have to pay that verification burden. In terms of, hey, I've got to go actually check that this-- the thing that I wanted it to do rather than just looking at the code. The other bottleneck is the code review. Suddenly, you're shifting this-- what an engineer is spending the majority of their day doing from writing code to reviewing code. And there are very few people out there who would have said, I think, in this prior world, that they'd love code review as the most important part of the job. But it is something that is needed and something where we share best practices, share culture through. And now people were just doing it day in and day out. Because I can turn out 20 PRs in a day using a local agent. But if we have a policy where every PR needs a human review on it, well, then my team needs to go review 20 PRs. And yes, then they'll also be turning out a bunch of PRs. And then I have to review their PRs. And we just end up in the cycle of, we're actually spending more time reviewing. And then PRs get still. And then they have merge conflicts. And it's this other bottleneck that I think we never had to engage with before because we just weren't making enough code before. So it's very interesting when we compare that previous approach where Tesla was essentially using those coding agencies as an accelerator to writing code. Come the Friday, we're essentially trying to get as much done as we possibly can looking backwards on that week. Whereas today, with the Dart factory, what we're actually doing is it's a bad Friday if we don't prep the weekend. We're actually looking forward to think, can I plan and stack and queue enough for the Dart factory to have enough work going forward? It's a very different mindset in terms of you're essentially managing the agent work versus essentially feeding on a minute to minute basis the agent with tasks to do. - Yep, yeah, definitely. And it's a little bit of that shift in my mind. Like you're shifting up the abstractions, Acabit. You're no longer thinking about like what is this ticket that I'm going to try to get done. You're thinking about what is this feature that I want to do, what is the sequence of steps that someone needs to do it, just like I divvy up to a junior engineer or an intern or something along those lines. And then how do I make sure that the stuff that I care about, either like the correctness or the UI verification or the clicking the buttons or hitting at the endpoints or whatever it is that kind of eye is important to me, how do I make sure that's encoded in a way where I trust that it's going to get done and the agent is going to do it. And so I think it's both for me kind of like that stacking up in that forward looking aspect of it. But the other big shift has been kind of what we now have these agents, these models, where we can literally deterministically load context into their context window to enforce the fact that they've read something. It's very hard for me to get a human to go read a doc on best practices or principles or something along those lines. So I have this additional leverage from being able to say, you know that I want to build the code base in this way. So you're going to go build it in that way. And I therefore have a higher level of trust in the code that's going to be outputted. I can then also start layering in other bits of verification around like coding styles or around testing or around enforcement or stuff along those lines. And this is all about to that if you can cue it. The only way that I would trust to be able to build a queue that will then sequence and run autonomously is if I have enough trust that the output of that queue isn't at a bare minimum going to break existing functionality. But ideally, it's going to be something that maybe not proud of writing myself, but I would be okay if I had to write it in myself. - Yeah. Let's jump through what a Dart factory is component by component. And I'd love for you to walk us through how we started on this Dart factory journey, what we implemented first and, and yeah, really understanding that path so that others can kind of follow and understand what they should do first. So when we first tried to build this Dart factory, what were the initial steps that we took? - So I think they're just kind of set the level and a bit. There's a couple of important concepts here. One is the orchestrator, which is what's actually picking up tickets and then passing enough to coding agents, doing work and doing the whole babysitting thing in terms of making sure the PR goes through and all that kind of stuff. But then importantly, and I'll touch on the second is the actual context and environment within our repo that we give to the coding agents, which I personally think is where the higher amounts of leverage comes from. But we built the first thing first, the orchestrator first, just because that is the tractable thing, that's kind of what we think of when we think of a dark software factory is the mechanism to go from linear tickets to actually produce code. I think just one of our big learnings previewing this has been that is simple and not that hard. It's everything else that surrounds that, where you then, because if you don't kind of invest in those other layers, you then then end up back in that previous workflow case we were talking about, where you just end up over one with review, or you end up over one with code, or you end up doing things that aren't actually driving the whole system forward. >> So the orchestrator isn't here, an agent orchestrates it. It's essentially almost like a workflow orchestrator. >> Exactly. >> It's connecting, it's an integrator between agents and other tools or other things like linear as well as the good example of it. >> Exactly. >> And so Ardern started with basically a Python queue that connected to linear and connected to GitHub and ran an agent locally in a Docker sandbox. We literally have one of our engine on it to close our laptop for about a week because it was all running locally on our machine. And we had a much more adoption on that than we had expected internally there. And so kind of the design decisions that we made around that were trying to keep things as simple as possible and relying on other systems as our source of truth where we could because we didn't want to have to reinvent the wheel there. So we've made an opinion a choice. Linear is our source of truth or to be done. Everything must flow through linear. If it's gonna result in a durable artifact that is a PR in our code base or a notion doc or any of the other artifacts that are produced by the dark factory, that just allows us the audit trail, the debugging, the visibility on that. And it means that we can then rely on all the work that linear is done in terms of blocking relationships, sub tasks, tagging for the reporting and the mechanisms that we need there. And we don't need to build it ourselves. So we have basically a poll. We're working to shift this over to web hooks. It just hasn't been a top of the priority list because we haven't broken linear yet with our polling other than our tests, which we'll mention later. So we have something that polls linear for new work delegated to our factory in state to do. Picks that up, puts it onto the queue. There's a bunch of prioritization logic depending on kind of which project it's coming from, what the priority is, whether it's being blocked, stuck along those lines. That then gets farmed out to a coding agent. As I mentioned originally, that was just running in Docker on someone's machine. We just really want an isolated environment because we don't want these agents to kind of be interacting with each other because each one is meant to be kind of a one-shot implementation. Mind you, they can technically understand what other agents are going on. And we do see this in the logs because we'll often reference other tickets in the description of a ticket. And so an agent will go check in on how that ticket is going or what the PRs was associated with that. But that's kind of an emergent behavior in terms of building up the context needed to be able to do effective implementation. These agents currently for us run in a Daytona sandbox. We're using Daytona just because of the isolation and security guarantees there. This runs in our own internal Daytona environment. And we provision a relatively beefy box for that, technical AVM, in order to-- or technically a container actually-- in order to allow to spin up the entire Mono repo testing stack. So basically, everything that a local engineer can do on their machine, modulo creds that the local engineer has access to that we don't give to the agent, we want the agent to be able to do. This allows the agents to run our full test suite, including integration tests, spin up the Mono repo, click around on the UI, click around on basically anything that we would do be doing for validation and testing, all in their own virtual environment. That helps us build kind of confidence there. One really interesting benefit of that is that we've now been able to have the agent post videos and screenshots back onto the PR so that we, as human, agents can go manually verify the work without needing to actually SSH on the box and go click the buttons or cells. We can see what the flow looks like from that video, from that evidence, which helps them build trust and build confidence that it implemented the right thing. That is kind of the crux of the orchestration flow. The other bit of the orchestrator is just basically babysitting a PR. So once the coding agent is done, pass a couple pre-commit hooks, pushes up a PR. We're relatively confident that PR at least like passes, lint builds, passes tests, like it's a good agent for shot implementation. On the PR surface, so this is now where in GitHub, we were on two different coding agents. One is code rabbit, one is our own internal code review agents. Our internal code review agents kind of is anchored on skills. So we've written three at the moment. Other ones coming soon, skills that are targeting different bits of kind of the way that we, what our code review philosophy is. So we're on our own security, one around legibility, which is really is this code easy to maintain in the long term, and one around kind of catch all functional correctness. So we run our own code review tool. This is what's exposed via Tessel change review. And we are using that as signals on the PR for the agent to go fix. So code rabbit and the internal code We'll post comments back on the PR, and we have failures in CI when some tests don't pass, or we do some what's called mutation testing, property-based testing, a whole bunch of that kind of stuff, if those don't pass, those all get posted back as signals on the PR. The orchestrator is watching the PR, and so we'll re-wake the agent when those comments are passed back, and the agent then goes fixes them. It will take in the comments, it will reason over them, it then responds in line in those comments where appropriate, or as a top level comments, resolving where appropriate as needed, and also potentially escalating back to human for review. So it has the ability to say, "Hey, I need context, "can pause everything and go post that question back on one "year ticket, which then notifies the human." And so that is kind of the flow in a nutshell. Going from a linear ticket to orchestrator, to putting agent on box, to PR, to a bunch of cycles on the PR review between automated bots, and CI checks, back to the agents, until it gets to a state that is good. That's normally when we flip to in-review, and we'll get the human in order to engage with that, because it's kind of like, in hopefully the best state that it could possibly be. And so human, the interaction mode there is twofold. One, we do have the escape hatch, if they can pull down a branch locally, make changes themselves, and then push it back up. That's always an option, if it's gonna be quicker, or they need to do it that way. What we're trying to push for though, is doing the interaction mode on that PR surface via comments, because that then allows us to kind of track that, and also integrate that back into kind of what a review philosophy is. That makes it surfaceable, actionable, so that we can kind of continue to leverage agents, or continue to leverage those insights, into better review flows, so that we don't need to get a human, to identify that stuff again in the community. And that's just for this. - And there's pure requests that do require a human interaction. - Exactly. - For a review. - Exactly. - And the administrator in this sense actually does sound like, kind of what a human would do today in that kind of flow, so it's like, and let's talk a little bit about each step there, because if we were to ask an agent to do various things, whether it's coding, whether it's design, whether it's a bit of architecture work, or whatever at the various stages, it's not always gonna get it right. - What an option. - How do we manage to ensure that either the agent, or the orchestrator, is validating that what is being built, this is pre-tests, what is being built, is being built to our guidance or our standards. - So twofold. One is we intentionally shift task ownership, and task the composition onto the human at the moment. So we're not saying, give the dark factory your Q1 plans and it will go do all of that for you. We're saying give the dark factory a well-scoop linear ticket, like you would to a junior or mid-level engineer, and it will go implement that for you. And so there is an expectation that there has been a fair amount of design work, design thinking, potentially white boarding, potentially discussion beforehand, which you are then able to encode in the linear ticket, which then kind of guides the agent ticket implementation after that. And so think of it as your job no longer is take ticket and go implement it, your job is have the white boarding conversation with your colleagues capture that either as like a granola transcript, or a screenshot, or whatever, and then work with an agent to then translate that into a set of tickets that the dark factory can then go implement for you. So there's a little bit of that shift of prior work up front. In terms of the actual, does it follow what you do? There's one thing is implementation is free, or functionally free, you have to pay token costs. So kick off six, kick off seven, like you've got a couple ideas, just go kick them off and don't send them an auto merge and go look at the code. Like I kind of almost view the fact that there is a potential like wide state space of potential solutions as either you don't care about it, and so you should accept whatever satisfies it, or be your unsure and so you should use the agents to then help explore that state space, and you can round your analysis in a real code artifact, like how it looks and how it feels, rather than just in the theoretical abstract of, I think this is going to feel nicer if the interface is designed this way or that way. And so that kind of approach, I think is quite powerful because it's allowing you to kind of do more experimentation, because you don't need to run the experiments, Darfactory does it for you, and you're just kind of looking at the outputs of that. On the taste and the architecture specifically, our solution to that has been verifiers, which are basically kind of like natural language, single statements that can resolve to yes or no to a Boolean about a specific piece of code. And so they're the mechanism that we're giving engineers to kind of instill principles about the code into their own specific parts of the code base that will then only trigger when those parts of the code base are modified. And so-- >> And so the deterministic trigger? >> Oh, that's-- >> It's a deterministic trigger, but it is sent to an LLM as a judge in order for the Boolean evaluation. So think along the lines of something like these files should only ever import from other library files. Yes, you can technically write a very convoluted regax in order to do that as a deterministic link rule, or you can encode that principle of like, hey, we want library to be the single source of truth. And so things should only ever be importing from the library and should never be importing from other modules within the code base in order to kind of maintain clean dependencies. And so that kind of taste is really hard to write deterministic rules for and we've historically kind of caught in code review, agente code review is expensive and slow. Can we shift kind of these discrete chunks into a human-legible form that we can then put next to the code so that the agents can go being their head against the wall in terms of making them pass as part of the CI flow? So I think that's a little bit of the shift in framing as well. >> And it does feel like those-- I think you've alluded to that. The verifiers are very, very fast to run. >> Yes. >> In terms of the task that the agent is trying to do as part of that verifier, it sounds like the scope is much, much smaller as well. So for example, if that one check was part of a code review, it feels like that's going to be missed by an LLM because not all the time, but that's going to be missed by an LLM some of the time because the code review covers a huge number of things. Whereas if it's a small scoped piece that an LLM just needs to check on this file or on this set of files, this one thing, it's going to have a higher chance of actually validating whether that's done or not. >> Correct, yeah. >> And so a bit of our philosophy is, agent code review should be the catch all for things that we miss. But once we have identified something that we think is important, we should be shifting them to verifiers if we can, because as you mentioned, there are a lot cheaper to run than to an LLM call over the diff. And so we can assess that. And we also run one LLM call per verifier. So that LLM is really just reasoning about this single verification, not about all of the conflicting priorities that it might have in its head. So we're trying to kind of sketch out a boundary around the code, around what is acceptable code from a bunch of different points, rather than trying to build a smooth line, basically in a skill or in an agenda review or something on those lines. The other kind of principle that we have is anything that is a verifier that we can make deterministic, fully deterministic, well then we should make that deterministic. I think historically a lot of people haven't written really complex lent rules because they're annoying and finicky and they slow human engineers down in their hearts, right? We're well right now, we have no human engineers. Agents are fine with annoying and finicky. They can very happy to go bang their heads against the wall in order to make something work. And they're not quick, but like an agent can write it, it doesn't matter to me how convoluted it is. And so that then allows us to not even pay the LLM cost, then we're just playing the CPU cost of kind of deterministic verification that we're good. And so it's like a sequencing almost for engineers to kind of build their, the things that they had in their head around quality or around what good code, look like or how good code smells. We can kind of put that into the code base in that sequence of the very, very quick deterministic verifiers, linting rules, the LLM is a judge verifiers that are just over the diff, which are natural language. And so probably too fuzzy to express, but easy enough for a good reasoning model to understand. And then the catch all agent review. One of the really interesting things about, I think about that sequence is that we can run automatic promotion. So every night we look at kind of the PR comments that are being posted by agent review and human review, and we assess whether any of those should be verifiers. And so we can kind of continually shift left in that regard to get faster, to get quicker, to get cheaper, but also importantly as well for me, I think, to get more human legible. If I look at a verifier, it's much easier for me to reason about that, than to read a long skilled on MD and figure out, as this thing going to be correctly weighted for something that I think is really important against all of the other compounding tradeoffs that it might have. That's super interesting. Let's go into what worked and what didn't work. Obviously this unlocked, it sounds like a ton. What were the biggest learnings you had early on with this? Coating agents are really good, and if you give them a wide enough scope, they will do some great things. I think there was definitely a bit of a kind of bias internally going from, kind of going from a philosophy of, I've got to be there babysitting the Coating Agent, to shoot on it. I can't even look at the logs for this, because it's running on a sandbox in the cloud. I better think that my tickets are super well-scoped and super small and super tiny, kind of like intern level tickets. And now we've slowly been, and this is cultural, not technical, slowly been relaxing those constraints. And people have been like, "Ah, cool. What is actually the biggest thing that I can give to the dark factory and see it produce something that is relatively valid as a result?" And so the tickets are getting trust? Was that a trust issue? It was pure trust issue. I think trust, compatibility, seeing the results. We're asking people to trust that something that is kind of hidden, it's called a dark factory, because we're not looking at the logs, as it's kind of operating. We are post-doc, but not kind of in the actual operations, they need to be able to trust that that's going to do something that they would be okay with. Because ultimately, we still have an accountability mechanism. The person who filed the ticket is still accountable for the code they got produced. They still have to, if it's in the production part of the code, but they still have to review the PR. We're not saying, "You cannot obligate all responsibility because the dark factory did it." And so we're now asking people to assign their own accountability to something that depends on us to produce it. So there definitely is going to be a gradual kind of trust adoption curve. And we will continue to go through that. And I think it's going to be never ending because there's always going to be kind of an increased scope or increased verification. You can do to build that trust. Yeah. I like to think autonomy is earned, not enabled. You can't just turn something on and say, "We can now go at full speed." It's about building that cultural trust internally in an organization to get to the next stage. What problems did you have there? So just on that trust bit, I think one of the things that we've been trying to do on the team to build that trust is by running as far ahead as we can and running into the walls early and mucking up, basically. So we definitely, that's kind of you, if that's the case. What are the wolves we read? So I think the most interesting, there's two kind of classes of ones, one around context and one around verification. In terms of context, I pull data for the number of SlackFords on-cent on our team channel versus the other team channels kind of every week. And our team, despite being slightly smaller than one of the other teams of the company, is sending about between 10 and 15 X as many Slack messages per week on our team channel. We have had to over-communicate everything that we're doing because we're doing so much more quicker. And also with Auto Merge on, everything is changing in ways that we don't really understand. And I'm not, I don't think people actually go look at code anymore because they're all just using agents. And so we don't know what's going on there. So we need to build the signals in the context on top of that to give us trust that this is working the way that we think it's going to be working. But then also we know what everyone else is going to be doing. And that 95% of the dark factory codebase. Yes, of the dark factory. Specifically. Yes, everyone has looked at the production codebase. Again, we have human review required there. So that has been looked at. But it's for the dark factory itself. And so we needed to figure out mechanisms in order to make sure that everyone knew what they were doing, make sure that everyone was comfortable with their own work getting through and away, and make sure that we understood how the codebase was changing over time. And so we definitely still have been figured this out. But I still will wake up on some days and be like, I don't think that's a good idea. Why did that land? But the intention what we're trying to do there is overshare the context so that we're aware. But then also fix forward. It's not roll back up here because it wasn't the thing we wanted to do. It is okay, cool. What we missed on the verification side that allowed this to land. Either in a verifier if it's a human concept or on tests if it's something bad, so that we can continually fix forward and build those guardrails, earn that trust on the autonomy side that you mentioned. In terms of some of the more spectacular failure modes, as part of this, we changed our core queue a lot in the early days because we weren't verifying it strongly enough. And by what I mean by that is we would have a fix, actually concretely, we had an issue where PR comments were being double counted. And so they would end up on the queue twice and so you would get an agent in two different sandboxes responding to the queue. You could have race conditions where my produced two different code comments is part of that. The second sandbox would start up after the first one I would be committed so to recognize hey, the work's being done, but it wasn't very effective. And the queue here, just for context of what the queue is, the queue is the list of tasks that the orchestrator needs to pull to then pass to an agent or do something with or something exactly exactly. And so we would fix that. It'd be good for a day, then it would regress, then we'd fix it again, then it would regress, then we'd fix it again. There was a point where I think we had like 60 PRs over two or three days that we're all trying to fix this. And so this was just causing constant churn and frustration because we'd say we fixed it and then it would regress and it was just, it was a pain. And so the way that we wanted to try to fix that because we were still going to continue to let agents do whatever they wanted was really leaning into the verification side of that. And so what we ended up doing was building a quint formal model of our queue behavior, specifically the in a potency of being able to kind of add stuff to the queue so that we could verify that on every single PR that we had been broken that formal model. That took, I wouldn't have known how to do that in pre-agent world, just flat out, I knew what a formal verification was, wouldn't have known how to actually build it. It took me about a day with Fable and then using a dark factory to go implement this and we've had absolutely zero recurrences of that after the fact. And so that for us was very painful in the moment, particularly as we were scaling up, but then be a really important insight of like no, it's not just the kind of verification that we used before, if like test and shared human context and like trust that a human has enough context about everything else going on to know that this is a sensitive part of the code base. No, we actually have to lean in even firmer than we had before because agents don't have that contextual information and we don't want to give it to them because that burns to tokens, which is therefore more expensive and slower. So that was kind of one really interesting annoying, frustrating bit of time. The other one was an experiment around during the off-site, which I was really excited for in which really did not work the way I wanted it to, unfortunately. And that was explicitly stress testing this verification layer. My personal belief is that if we get our verification layer strong enough and if we are oriented around specifically the behaviors that we care about as an organization, not the actual implementation of the code, I don't care. I don't care about if you how when I click a button on the UI, the results appear. I care about when I click a button on the UI, the results appear. And so we should be able to just from that verification layer be able to rebuild the entire system from scratch in a new language. So we tried to do that. We currently are Python implementation. Elixir is ish a good fit for this in terms of having kind of the actor model. And so we took our verification layer only. So none of our unit tests, nothing other than kind of our NZN tests, our integration tests, this formal model that we were talking about. And some of the properties that we verify kind of on the surface boundary rather than on internal dynamics. And we gave that to a coding agent to then go plan out and building that in Elixir from scratch. And then we went into that. That worked. It built something. But we hit kind of two major roadblocks that kind of prevented us from being able to call this as a success. One thing I will note, our key was perfect. He was totally fine. The formal model held up well there. But what it ended up identifying was even after kind of all the investments on verification we thought we had made, we had some pretty big blind spots. We weren't actually verifying formally anything to do with the labels on linear, which are very important routing mechanism. That was all just built into unit tests and our N10 tests were just testing the happy path there. And so when we ported it over the N10 test, oh shoot, suddenly the way that we were stacking PR is broke because that wasn't tested in N10 test. That was just in unit test. So that's core functionality that wasn't verified externally. And so it was broken in the Elixir implementation. We weren't able to kind of push out to production, even though we run in a sandbox production environment. So then I had to mainly babysit an agent to go do that. And so that's an area where we want to be investing in in terms of getting the dark factory to be able to make non-destructive changes in production environments, or at least the sandbox environment first and then eventually production environments to take the human out of the loop there. And then finally there was just some other bits of the functionality. I mentioned the stacking of PRs in terms of the feature branch. I mentioned labels. I mentioned, or rather there's also some of our GitHub functionality specifically in terms of not just comments, but on CI batching where we try to take in all of the signals on CI failures and send them as a single as a one off rather than sending them as a sequence that isn't captured by an N10 test. So there was some pretty core functionality and improvements to the system that weren't being captured by this verification layer. And so despite really wanting to turn it on, well I did turn it on and then it immediately broke. And then I fixed it and then it turned on again and then it broke some people's PRs and I had to be like, okay cool, this didn't work. And so I think the real learning from that is this, I think is doable. We got pretty far with it. I think it is an even more ambitious framing though than what Anthropic did with the Rust Re-Rite where they had kind of the original Ziggs source code as an input to that. But I think it actually allows you to write idiomatic code better because our Eglixer implementation, I'm not a look sir expert, so this is entirely LLM as a judge agent vibe. Our look sir implementation evidently was idiomatic elixir. I don't really know how to assess that but evidently so rather than being Python 2 elixir ported. But we need to make sure that we cover everything. And I don't know yet exactly how we're going to continue LLVR up on that other than by continuing to do experiments like this to be able to identify where the gaps are because that's the only way that we can build trust and really let agents rip when we're confident that that boundary which we do require a human review on is super solid to the extent we can rewrite it. So one of the things I'm getting here from this discussion from this conversation is that in building this dark factory this isn't something that we can necessarily just turn on this. expect everyone to use. A lot of our learnings have come through usage and I'd love to kind of like switch briefly to adoption and onboarding. So, you know, when we would be able to now with the Dart factory that we've got now, if we had a new team that had never used the Dart factory, do you feel like they're better just like on day zero, just switch in directly to a scenario where they're using a Dart factory or do you feel like they would need to grow into it like lab by lab piece by piece. So, we had a new starter who on day two was adding verifiers to the Dart factory, which as I was just saying is one of the most important parts of the Dart factory. So, yes, I think you can get new people onboarded to it, but it does require a specific mindset to. I think it needs to be folks who are recognizing the fact that agents are not just doing the same thing but faster, but are fundamentally new workflow primitive and how we do software engineering. It's this shift in my mind of like what is the role of the engineer? I personally don't think engineers are going to go away. I think there's always going to be need for technical expertise in terms of understanding what the systems are, but just as your mechanic doesn't build your actual car, but can go debug what happened with your car when it breaks, that I think is going to be more so the role of the engineer in terms of helping build and create the environment for a factory to run in and then debugging why it broke and then improving that. So, more around the outside and that I think for kind of a new starter or a new person, it's definitely a mindset shift, but it's still engineering, just engineering at the systems level, which is what you were meant to do as a senior staff principal level engineer anyways. We're just now asking junior candidates to start doing that right away. Hey everyone, hope you're enjoying the episode so far. Our team is working really hard behind the scenes to bring you the best guests so we can have the most informative conversations about agentic development, whether that's talking about the latest tools, the most efficient workflows or defining best practices. But for whatever reason, many of you have yet to subscribe to the channel. If you're enjoying the podcast and want us to continue to bring you the very best content, please do us a favor and hit that subscribe button. It really does make a difference and lets us continue to improve the quality of our guests and build an even better product for you. All right, back to the episode. So, an individual can join an existing team that has all of this set up for their project. What about an organization that is trying to, that is, you know, perhaps where Tesla was a little while back and not using a Dart factory or software factory and perhaps using the Claude code and codecs and things like that to accelerate that building of software. Is that something they can snap to directly like we're going to turn on Dart factory and start building this way or is that something that should happen gradually in layers? Yeah. So, I've been talking a little bit about what we're doing with building the Dart factory, but as I mentioned, like we intentionally tried to front run a lot of these issues here. What we've been doing with the rest of the production code base is that kind of incremental adoption because we're not going to come out and say, hey, everyone at Tesla, you must use the Dart factory. We've got into that kind of 65% or 6% of PR is going through the Dart factory by convincing people that this is a fun new cool way of working that speeds them up. And that is very much so by helping them lever up the parts of the code base they own through all these tools and mechanisms that we've identified specifically around the verification to be able to allow them to build that trust and to build that faith and automation. So, concretely, what that looks like is just start with better context in repo. Like that already improves your local developer flow. Like people, companies, organizations, I think really need to be much more intentional a bit the context in the repo because that is ultimately what goes into the agent, which ultimately in my mind and part of my kind of belief of where the value is that we can provide more important than the actual world underlying model. The context that you give will change model results more than swapping a model will. And so, if you feel you're at a good place in that regard, then start layering on the verification bit because that then also helps you trust what agents being driven by local engineers are producing because they are verified two ways that kind of go beyond what you might have done historically. So, test or debt cheat, debt cheat for right, right, test for everything, right end to end, test for everything, right integration, test for everything, do behavioral tests for everything. Make sure that that kind of layer of the thing that you actually care about, which isn't just what the code does, it's the behavior, it's the product surface, it's things that is expressed. Make sure that is super solid because that mechanism will both help build an adoption kind of from your internal engine because hey, your agent now is producing better work without you needing to say kind of the reins and guide it through that as well as fully producing higher quality outputs over time. I think then you start layering on into the more non-deterministic aspects. So, adopting verifiers, adopting these kind of discrete units, which are still very human, reviewable, very human, understandable, but are enabled to encode kind of fuzzier concepts than you would be able to and deterministic limiting rule. Again, you can then have something deterministic run that, feed that signal back into a local agent or into a dark factory and use that to kind of encode preferences in the code base and then into a genetic review and then kind of levering up in that way. And so, these pieces that you can adopt particularly around the verification side, these both do better when you're kind of or rather allow your engineers to do better when running local flows and give you the building blocks and then move closer and closer to fully all night of factory. The last point on my mind just around the cultural adoption bit. I think the most important thing we've been able to give to the engineers here at Tesla is the feeling that they still own the taste and the quality in their code base and they're not just giving that up to an agent, but that we're trying to help them figure out a way to scale out their taste to all of the agents that will be running on their code base. Previously, let's say you were a staff level engineer who was responsible for a specific part of the code base, the way that you would enforce kind of quality there is doing all the PR reviews or doing presentations or writing docs on what coding standards look like or writing a style guide line or stuff along those lines. Now you encode that within a set of verifiers, your agents, all the agents, not just your agents, all of the agents of everyone in the company is going to be running that to those verifiers for those bits of the code base and suddenly your impact is now scaled across everyone. That then forces those agents to engage with it and also allows you the ability to take your learnings over however long you've been in the industry and give them to the agents in a way that makes them better. That I think is where the truly scalable shift in mindset is coming from. It's not just what can I do to make my local agent better, but then can I commit that into the repo in the form that makes all of the agents better and also then gives me the base to build up towards a dark factory future. Now a lot of people are using Tessle today. Using that as the registry to store their context to evaluate and improve their context, distribute their context across organizations in private and public workspaces. But the Tessle agent of course, why it can be a great interface to a lot of those functionalities and capabilities. The Tessle agent can help here as well in doing a lot of what you just said, right? Yeah. We think of the Tessle agent as a natural language interface for being able to build the building blocks of a dark factory. It has knowledge of all of the bits within the Tessle CLI that we touch upon here. When I was talking about our own internal could review, that's Tessle Change Review. When I was talking about verifiers, that's Tessle Change Verify. We also have Tessle Change Risk, which is our own internal risk implementation to assess whether PR needs human or view or not. All of these core primitives are baked into the Tessle agent and the Tessle agent knows how to set them up in Erepo. As I was mentioning earlier, we don't think the orchestration layer is differentiated. Everyone probably should build their own orchestrator because it's simple and allows you to then integrate against a whole bunch of your own internal systems. Do you use Gira? Do you have your own internal custom ticketing system where your source is knowledge? That is important to work, but the differentiated work in my mind comes from the loops that you're able to build on top of that and also these primitives that you're able to compose. Tessle Agent will then walk you through how do can I set this up in Erepo using your context using your way of doing it? For open source repos, we recommend using GitHub Actions. They're easy, they're cheap, they're everywhere. We can then Tessle Agent can then work with you. To be like, cool, I want to do a review of all of the PRs that have landed in Erepo every single week and derive verifiers from those. Tessle Agent can do that with you. I'm being key there to say with you not for you because that verification bit just to reiterate is something that I think where it's the human leverage comes from. That's still very much so part of the core engineering job because that's where you're including your taste, your preferences, all that kind of stuff. You're still in control, which is key. Exactly. But the actual like, go right to GitHub Actions do it. I don't want to do that. Tessle Agent will do that for you. And if I wanted to get started, I've got a fresh project that I'm building. I'm currently, it's an energetic code, I'm currently building using various agents. I've got various skills and context in that repo as well. I'll start up Tessle Agent. Where do I start? What's the first command? What's the first prompt I should write? Cool. So I guess two commands come immediately to mind. One is do not out of your code base. Just ask it to help with that in terms of being able to understand what context you have, where are agents thrashing. We have the ability in there to also go look at some of your local logs if you are using cloud code for example, it knows where to go find your logs for that session. This stays all local to your machine to be clear, but it will help you be able to identify like, hey, this is something that the agents have done that is constantly thrashing and constantly kind of causing issues. Do you want to write a skill for that? And then do you want to kind of be publishing and versioning that skill? And so that kind of gives you an initial set of insight into like where current friction points are or where you're already great and you don't really need to do much. And then the second one is on those loops. It's like, what is the first loop? Like get the agents to suggest you a loop. My personal bias would be probably something looking at the PR surface or something looking at like docs updates or maintenance. You know, there's kind of tasks that you know you should do, but you never want to do. And so you get still docs. Agents and loops are great at that. And so working with the intelligence agent to figure out kind of what the highest, either highest value loop is for you to still then be human loop or highest value that you're comfortable running autonomously because it's a really interesting and cool feeling to be able to wake up and see, hey, we call them maintenance agents. And you're, certainly, hey, my maintenance agents ran overnight, identified three issues. File PRs are all on an on-emerge and they even landed. And so that just is great to seeing work done. Exactly. And it's not worked that I had to think about. It's like, I even have to initiate it. Yeah. It's like, I found the signal and I want the signal to go up or go down or do something. Yeah. And then maintenance agent goes and does that too. So that's available in Tesla agent right now. So if you have the Tesla CLI installed, just type Tesla agent on the CLI on the terminal and you'll drop into that environment and you can ask those prompts straight away. I think it's time. So we go back and have a look at some of those linear tickets that we raised. Let's see if the dark factories done something with them. Definitely. This one, as a reminder, is about adding the metrics for where we did the inference geo within the dark factory. And so we can see that, hey, there's been a PR associated with it and hey, that PR has been merged. So I'm just going to click over to the PR here so we can see what happened. So we've got the summary from Kikimora. As I mentioned, that's our internal name for our dark factory. We have CodeRabit who runs on review. Disk GitHub Action here is our own individual review harness. So these are the comments that have been responded to by Kikimora and resolved given by our own action. CodeRabit also then provided some comments as well. There are some CI failures. Thank you, Kikimora for fixing that. You can see that CodeRabit approved it and Kikimora merged it. So this was, no human has looked at this up until the point where I'm looking at it now. This joins that 95% of code that has been merged with no human exactly that in our dark factory. Exactly. You'll look at it because I trusted the test pass and therefore it at least isn't broken. And if it is broken, we will find that out and then we will fix it. We'll fix it forward. Absolutely. Awesome. So this was the first linear ticket that didn't require the human interaction. Perhaps we jumped to the linear ticket that did require that human interaction which would go into production. And so this is the ticket where we were looking at fixing the quote issues in terms of what was landing on our web UI. So again, something that was identified in an audit but probably should be fixed and something that is fully user visible because this is the code or this is what user see on our website. We can see that there has been a PR associated with it and it is also merged. So clicking over that, we see that 46 minutes ago, so 10, 15 minutes after we get the ticket off, that's when the initial PR was opened by Kikimora or internal name for the dark factory. We see code rabbit go kicked off as well as our own internal code tool. A bunch of issues found Kikimora then responds to it. Code rabbit approves it. Then Seheal, who is one of the engineers on our team, as I mentioned, these require human review because they are production touching. He then approved it and merged it. So the only human touch points on this PR to get something from linear ticket to production is that required human review because it is still production code and someone clicking the merge button. So the fact that Seheal did that was because this was flagged or this was marked as something which requires human review because this is a critical system, maybe a production system or it might not be a production, but it might be some critical piece of code that we require a human to look at before. Do you feel like there will be a stage where we will fully trust this? Yes, there will. I think so one of the things that we are looking to invest more in in the near future is how we do risk classification on PRs. I would love us to get to a point where we are able to merge straight to production for low risk PRs according to that classification. And the way that we think about that classification, like a lot of else, a Tessel, is what is the human visible mechanism for being able to encode that so that we can then reason over it rather than an opaque classifier model that's doing a binary classification? What is the skill we can give to the agent that encodes the things that we think are important to so that we can then tune that to that specific risk level. And so if it's something like this, probably would be out of March at some point in the future because it is a user visible, but it is just a documentation change. Doing a full database migration, I still think that one actually might be because that's easy to verify mechanically in terms of switching over what infrastructure our pipelines run on. That one might require some human in the loop, but I think we are working towards that goal. So during this podcast, we created two linear tickets that have now had PRs created and have been merged, one into production, one into the Dart factory codebase. That's two of six to 700 pull requests that will be created this week at Tessel. I think we said we're at about 60, 70% of all pull requests at Tessel being created through the Dart factory. What will it take to get that to 100%. That's interesting trust, allowing shifting left. So actually, stepping back quickly, we think of it in shifting left and shifting right. So from the PR, shifting left is helping engineers create PRs. Shifting right is working on the verification of those PRs so that we trust what the output is. And we think we have historically focused more on the shifting right to build that initial level of trust of like, hey, the code the agent has done is roughly correct. There is always more that we can do and there's always more that we will do there to kind of improve the quality of the output. But we think we're in a good enough spot there where we really want to be doubling down and investing in is on how do we make this the easiest way to do work at Tessel. Not just the fastest or the highest throughput or anything along those lines, but where can we bring the Dart factory into making that even easier than possible. We have our own internal slackbought that listens ambitantly in channels and can file linear tickets for you and all that kind of stuff. We want to move or we want to move it into actually being able to propose tickets on linear or being able to do decomposition. We want to move it into the planning process to be able to help you go from PRD and be there as a coworker to kind of be able to refide the is off of or something along those lines. So making it so that the default is that our factory not because we're enforcing it, but because that's just what people reach to because it's integrated with the tools that they're using every day. Amazing. Rob, this has been truly, truly fascinating and it's an amazing accomplishment to see how much of an impact this has had on the way we built software at Tessel today. And for our audience, our listeners, if you wanted to try this out and actually build your own software factory, head to tessel.au for more information about Dart factories and about the Tessel agent. Load up the Tessel agent just by typing Tessel agent at the command line once you've got the Tessel CLI installed. Just ask it those questions. Take a look at my environment. What can I do to improve this and how can I add these verifiers, these plug-in reviews, the pieces into the CLI and the code reviews? This is absolutely amazing. It's definitely the next level of software development. Rob, it's been an absolute pleasure. Thank you so much. Thank you. It's been fun. I hope everyone learned something today and thanks for listening. Tune into the next episode. Bye for now. The AI native dev is brought to you by Tessel. The package manager for skills and context. Your hosts are Guy Pajani and me Simon Maple. Our producer is Tom Dowler. The AI native dev is not just a podcast, it's a community. We host monthly meetups at the Tessel offices in central London. Visit tessel.io/community to learn more and I hope to see you there. Thank you.