How Warp ships 2,000 PRs a month with AI factories | Zach Lloyd (CEO, Warp)
46m 51s
A software factory is a comprehensive, code-defined automation system that streamlines software development by integrating multiple tools—like Slack, GitHub, and Figma—into a single, transparent workflow. Unlike traditional coding agents, it spans the full software development lifecycle, from idea to code review and QA, with public, trackable processes that enable team collaboration and oversight. A key insight is that human bottlenecks, especially in code review, remain significant despite automation, but the factory reduces friction by standardizing workflows and improving efficiency. The system introduces self-improvement loops where agents analyze past failures and update factory configurations to prevent recurring issues. Centralized metrics—such as prompt interactions, PR velocity, and cost per task—provide engineering leaders with real-time visibility into performance and automation maturity. These insights are further enhanced through automated scoring, which evaluates agent outputs for quality, redundancy, and alignment with user needs. The factory also enables non-technical use cases, such as refining sales decks or identifying customer questions from meeting transcripts, showing its broad applicability. Ultimately, the factory transforms development from a siloed, reactive process into a measurable, scientific, and scalable system where both human and AI inputs work in harmony to deliver better outcomes.
Can I give you a hard time that humans really are the bottleneck?
Because if you look at kickoff to PR time, it's 35 minutes.
But if you look at PR to first human review, it's three and a half hours.
And when you're doing, I think it was like over 2,000 PRs in the last month.
Like, how do you keep things in the team from going as I say, like, chaos range?
What is a software factory? It's for us, at least. It's an actual noun.
It's like a product concept where it consists of a bunch of
repos, a bunch of like MCP servers, like a bunch of configuration, and then a bunch of agents,
essentially, so like a code of reagents is not different agents, different automations,
and it's all defined in code. Scoring happens across all runs.
But then there's a second loop popular thing on Twitter right now called self-improvement,
where you have like an observer agent and it can then create updates to your factory that will
try to prevent the particular failure mode.
One of the things that's most helpful is like you could have these factory agents
do computer use verification. So in this case, it made a video.
I'm just like talking to this thing that is doing this job that I've done for the last 20 years,
and it's now it's like doing it kind of better than me.
Spoiler alert, the humans are the problem.
Welcome back to How I AI. I'm Clarevo, product leader and AI obsessive here on a mission
to help you build better with these new tools. Today I have Zach Lloyd's CEO of Warp,
and he's going to show us exactly what he means by the software factory. He's going to show us
how you can kick off tasks and slack, what it means to go beyond the software development life cycle,
and how a technical CEO uses non-technical tools with AI. Let's get to it.
This episode is brought to you by DX. In a recent study across more than 500 engineering
organizations, DX found that spend on AI tools has grown 28X over the last year.
The share of AI authored code is climbing, but overall innovation has remained flat.
As teams generate code faster, new friction and code review and validation is offsetting
those early velocity gains. DX tracks speed, quality, and cost together across the software
development life cycle, giving engineering leaders clear visibility into how AI impacts delivery,
and whether those investments are translating into real value. Download the full report at
getdx.com/howiai. That's g-e-t-dx.com/howiai.
Zach, thanks for joining Howiai. Thanks for having me. Excited to be here.
There are a lot of trends on the timeline right now, but one that I think is going to be big for
2026 or seems to be big for 2026 is the factory. The factory, this fantastical idea that we have
that we're going to input tokens and output enterprise values, something like that. At least we'll
output code. I'm excited for you to show us as CEO, as engineer, as builder. What to you is the
factory and how it looks inside warp? The factory is definitely trending. I actually don't love the
term factory, but it is the thing everyone is saying. It feels a little dehumanizing to me.
The idea is exactly what you said. It's like how do you, in a world where we have these magical
agents harness their power in a more organized way to have them basically build software for you
from your ideas. Excited to show you how I use factories and what it means to me.
I was at Lenny Summit recently, and I think Marty Kagan also said he hated the word factory.
I wonder if it's more like Santa's workshop, right? We have magical elves that craft wonderful
things that spark joy and are delivered gift wrapped for us. I'm going to say the AI magical workshop.
No, show us what does this look like for you? You just say you put in your ideas, your output
product. Is that real? How does that really work? Yeah. It kind of works like that. The way
that I think of factories is there's basically two types of personas who are using them.
So I'll show you basically how we do this in warp. There's the builder of personas. The builder
persona is someone who is contributing ideas, wants to build stuff. And from that persona, it doesn't
look like it's the way that we have it said. It doesn't look that different from like how one of
these engineers or designers or product people on our team might build with a local coding agent.
The big difference you can see here, so this is like, this is our slack, is that most work is
starting in a public place. So the way we think of factories is like you basically can say in
a public slack channel, you can tag, we named our factory Wilson. You can tag Wilson and be like,
okay, I want you to build something for us. And when you do that, what Wilson will do is like
essentially do not just like the direct build, which is what you might get if you're using like
cloud code or codex, but we'll also like do a few other steps. So what what we'll do is it'll
kind of go through all the steps of like first it will triage whatever this is. So it'll treat
it as like an input to the factory, not just like the direct thing to build, meaning it will
you know, for this one Harry's saying he wants to change the way this feature looks. He gives like
pretty detailed instructions. He you know, attaches an image. He says use computer use and this kicks
off a flow where the the first thing that the factory does is actually kind of open an issue.
So you can see over here, it's like we use linear and addition to using slack so that everything
gets tracked. So the factory integrates not just with slack, it opens an issue. Then it does the
actual implementation. For this one, it's like it was simple enough and ambiguous enough that
we could actually implement it. Then it does, it will create the PR so it integrates also with like
GitHub. Then it does the QA, which I think is like really important these days. Like if you want to
not spend all your time doing code review, one of the things that's most helpful is like you could
have these factory agents do like computer use verification. So in this case, it like made a video
of the completed feature. So it's like showing the thing, showing all the keystrokes. And so what
you get here is is not just like, and then it gets merged, right? It's not just like the single
step of build the thing. Like if you were to do this with like the coding agent in like the
prefactory world, what you would typically do is like pull up your local coding agent, do the change
there, test it locally, and then like push it up to GitHub. But you get the whole thing of like the
ticket, the code review, PR, the video, all done together. And then the other like really
kind of magical thing is that it's all done in public. So if you are, you know, someone else
wanted to come in and like look at this and see how the task was done, or even contribute to it.
And so we will have people like multiple people on these slack threads, you get to a world where
as like a builder, you're no longer sort of working in this local silo, instead you're working
in a public space where everyone can see what you're doing. And so that's like a from the builder's
point of view, how they like use the factory. Does that make sense so far? Yeah. And you know, I think
what I'm reflecting for folks who are maybe still trying to grapple with, okay, like what's the
coding agent? What's a factory? What's the difference? I think what I'm from you is the factory,
quote unquote, is comprehensive, comprehensively designed to reflect your version of what the
software development life cycle should be. And so it's not just like idea to code to PR to push,
it is, okay, an idea needs to go through several steps. It needs to go down the conveyor belt of
product and into like the funnel of issue tracking. And then we need to code it and then we need to
QA it and then we need to have these very specific verification loops where we can take our sticker
and say quality controlled by, you know, number one, two, three. And so to you and the way you're
describing it, where it goes beyond a coding agent or a co-pilot is that it is actually designed
to take take the very specific and end product steps, not just the engineering kind of like input
output. That's definitely part of it for sure. It's like it does, it does more of the software
life cycle. So it does more of those steps. There's a whole other bigger part of it to me, which
so I showed you this from like the perspective of the individual builder. And if you were to go into
our slack, you would see like this is happening over and over like all day long, the people in our
team are building in this way. And sometimes by the way, it's not just like the engineers who were
kicking off work like this, it could be work that's being kicked off automatically. So for
instance, we have, I don't know if you know sensory, but it's like a crash reporting system. And so we
have signals when there are crash reports coming from like our terminal app.
that we try to automatically fix those
and those go into the factory too.
So can either be human initiated,
it can be initiated by like an external system
and it can be initiated actually in any of these tools.
So it's like you can initiate it.
Like we could have started this whole thing in linear,
we could have started this whole thing and get up
and so it's integrated into all the tools.
But the bigger piece of the factory approach in my opinion
is not like, it's not so much how the individual builders work
'cause it's not all that different from individual builders
who might be going into the terminal for instance
and using a coding agent to do something.
It's different, it's in public.
But what's really different in the factory approach
is that everything is like centralized in the cloud
and there's a whole other aspect to it
which is for not the builders,
but for the people who are like the managers
who are like trying to scale software development
on their team. - Finally, something for the manager.
- But everyone's been wanting, right?
Too much attention on the builders.
No, but in all seriousness, if you're like my concern
as someone who's running a company, it's like I want,
I want it's a super duper competitive market.
I want to see how quickly we're moving.
I want confidence that the way that we're building software
is actually improving over time.
I want to make sure that we're not wasting too much money.
And in the world of like every builder
has their own individual local setup,
that's very, very hard to get.
And so this is like an engineering manager's dream,
it's like a CTO's dream, not that this is the coolest thing
in the world, but it's like,
now I have the ability to sort of centralize
and see what everyone's doing.
So for instance, on this screen here,
it's like these are, this is actual data from our team
in terms of like how we're using the factory.
It's like there's a sort of sense of like
how automated is the factory.
What's our velocity?
How long does it take us to like ship stuff?
And so there's all these measurements.
- Can we pause real quick on your productivity
because I actually haven't seen this measure before
and we've seen a lot of measures,
which is human interactions per PR.
Just talk us through why that.
Because honestly I've heard that one before and I like it.
- You know, this sort of instinct underlying this
is that we are gonna be,
not to put this the wrong way,
but it's like humans are a little bit
of the bottleneck in terms of production itself.
They're also the creative force,
but in general, what companies want
and we're trying to build for companies that warp largely
is like how do you automate more?
How do you make things that can truly just be done
agentically, be done in an almost fully automated way?
And so, you know, the general instinct is like
the more times you have the prompt,
the more times you have to steer
or like cajole your agent,
that's like, that's gonna be a limiter on throughput.
This is where you start to get into like factory world.
Like this is like imagining you're running
like a Tesla plant or something.
It's like how many times do you have to stop the line?
And so, you know, we're trying to give
engineering leaders a view of like, okay,
how are you able to sort of make things more efficient
over time?
- And quick question, are those just because my mind,
I mean, I, you know, as a one time CTO,
I'm like, yeah, this is exactly what I want.
Are these interactions, do you think of these interactions
as like the prompts, the sort of like up front steering,
are you thinking about like, how many comments on PRs,
how many loops, like how inclusive
is this like interaction per PR score?
- It's all of those.
So because the, and again, I think this can definitely be
refined over time and bear in mind,
we're like trying to figure out
so what is the right set of metrics,
but this actual metric because the factory
is like integrated into all of your knowledge work tools,
it includes all of those things,
like how many reprompts and slack,
how many comments on linear, how many times
did you have to correct the thing in code review?
And so, it's like kind of like, yeah,
it's a proxy for how much work did you have to do
in order to, you know, get the thing to do the job?
- Okay, wait, I want to give you a hard time,
one more thing.
- Yes, please do.
- Because before we started recording,
you said like, how technical?
And I was like, okay, well, we're going to put on the CTO,
at least the CTO how right now.
Because if you look at kickoff to PR time, it's 35 minutes,
but if you look at PR to first human review,
it's three and a half hours, three and a half hours.
And so it's like so funny, that's still cycle time.
Man, like it's the thing.
You really have to focus on, it's that number.
- Yeah, there's still the bottleneck, and we're,
I mean, we're still doing human code review.
- Do you do all your PRs get human code review?
- Currently, all of our PRs get human code review.
- Wow.
- And so, now this is like, I think of a team choice,
an organizational choice.
We have, the one thing that we have changed around this
is that like we used to require,
like the workflow used to be like person A on our team
would build something with an agent,
and person B would review the agent's code.
We no longer require that.
Like the person who prompts the agent
can also review its code.
- Okay.
- You're making me feel better.
- So that's a better thing,
but it's still like we, you know,
we don't have yet complete trust.
I think the way this will evolve is like,
some percentage of, some percentage of stuff will eventually
will feel confident enough that we can skip that.
- Yeah, I did an episode recently,
I built an EVE agent called Merge Mommy,
and this is the flow that I often see,
and even mature engineering organizations is what you do,
is you basically like risk score every PR automatically.
So go through risk score it,
and then anything that is low or extra low risk
gets a stamped approval from Merge Mommy,
and then a human's loud to just smash the button and merge,
anything that's like medium high or whatever,
or has some like outlier on risk,
requires human review,
and that just like lets you get that
bottom tranche of the queue out.
Also just so you have good capacity
for high quality human review on the things that really matter.
- 100%.
I think code review becomes an exercise in risk management.
I think that's right.
I want to show you some other stuff in the factory,
just to show you like how this is different
than just standard interactive agents.
So you get cost is on everyone's mind right now,
and so you get like, this is Asian cost,
it doesn't factor in the sort of cost
of the people at the moment, but it's like,
you can see like we were really expensive a few weeks ago,
we made some changes to our model configuration,
and we've driven this cost down,
and we want to continue to drive it down,
but just like having the centralized view for this
across your whole team.
- Can I ask you a question of cost?
- Quick question of cost.
- So I mean, this is a pretty,
you can see the drop here.
I see this a lot when I'm talking to engineering organizations,
they often see like a rise in cost to PR as their,
per PR as their adopting AI, which you all already have,
and then we see this drop as you're doing optimization.
Do you feel like the biggest lever here is model
right now, is that the lever?
- Yeah, it's model, I think model is the biggest,
it think like context, like the way--
- Context management.
- Like context management matters as a secondary thing.
I would say model is the biggest.
- Yeah.
- And the way to really figure this out actually is to test,
which is something I wanna show as well.
But basically, I think model is the biggest one.
Any other questions on the, just like the spend view?
- No, it's great.
- So the other thing,
like this is where it really gets kind of factory oriented,
is like the way that we're changing development,
and this is, I think this is most relevant for folks
who are watching, we're trying to manage
like these teams and scale coding agents,
is like we now, we're like measuring everything.
And so this last thing in here, which we call scoring,
basically gives you a view across all your agent runs
of like how they're doing on different dimensions.
And so we basically, because the factory is like a closed,
kind of like closed loop system, like every time an agent
does a task in the factory, that task gets recorded.
And what that opens is the possibility for you to go
and like go back retroactively and look at how well
the task was done.
And so you could do this as a person,
you could literally just go back and look at tasks,
but the other thing that you can do,
is you can have agents do this, which is what we do.
And so for instance,
Thank you.
if I can find an interesting one.
Like here's an interesting one, like redundant tests.
So if you're working with, you know what I'm talking about here,
it's like, oh, I know what you're talking about.
Every PR I push, it's like, I have run and set up 135 tests.
- Exactly.
- It's not, it's like, it's like, in the reason it does this
is because, you know, they're writing tests
not to like prevent regressions necessarily,
but also just like test behavior along the way.
And so you end up with a bunch of tests that you don't need.
And so we, you know, if you take the factory approach,
you kind of know this might be a failure mode.
And what you can do is you can write essentially a score
that uses LLM as a judge to be like, okay, I want to go
and look at, you know, all of the agent runs.
And I want to see how often we're, you know,
an agent thinks a different agent
than the one that did the task, thinks that there's,
we're done in test, this makes sense.
- Yeah, totally.
- And so you basically are like, okay, I want,
I'll pick a judge model.
This is like a kind of medium smart judge model.
You don't want it to be too expensive otherwise,
you end up spending a lot of money on scoring.
You have it classified in terms of like,
for this task, like, how did it look?
You pick like a sort of sampling rate
in terms of how you want to do this.
And then you get, over time, a set of runs
where you can see that like,
sometimes this agent thinks that there were some like,
surplus tests.
And so this is what I mean by like the real measurement,
then what you can do, it's like you can go
and you can actually, you know, as a human,
you can go and kind of look at what's happening here,
but I'll show you, there's also a better way to do this
in the factory world where it's like, you can actually
use agents to sort of identify what's gone wrong
in these runs and try to improve the factory.
Does this make sense?
- Yeah, I'm curious.
Do you do this on a per run basis
or on an aggregate basis across a set of PRs?
- Aggregate, great question.
So yeah, so what we do is like the scoring,
and there's what I call like this thing,
we call this like the score,
the scoring happens across all runs,
but then there's a second loop, which is another like,
popular thing on Twitter right now called self improvement
where you take an agent and you basically say,
okay, for all of the failed runs and you need like,
you need like a real sample, like maybe 20, 25 failed runs,
you need some significant sample size,
otherwise it starts to over-correct based on single,
based on single things.
You have it look, you have like an observer agent look
and it can then create updates to your factory
that will try to prevent the particular failure mode.
And so let's see if I can find one that's like,
I don't know, I'm just literally picking a random one here,
but this is finding some issue
with our skills that are driving the factory,
it's presenting evidence and it's saying,
okay, we should change the definition
of one of our factory agents in a particular way.
And if you go under the hood and look at this,
it's like, it's changing this like step 10
of what our factory agent should do.
This makes sense.
- Yep.
- And so this is, to me this is really exciting.
Like the way, the thing that's like enabling
this whole thing to work is that you define the factory
in code. - Code, yep.
- And so, you know, what I mean by that is like,
if you were to look at the sort of definition,
you would say okay, the factory,
what is a software factory?
It's like, for us at least it's an actual noun,
it's like a product concept,
where it consists of a bunch of repos,
a bunch of like MCP servers, like a bunch of configuration
and then a bunch of like agents,
essentially so like a code of reagent design,
like different agents, different automations.
And it's all defined in code
and the value of doing it that way with like an warp factories
is that you get the ability to actually,
you can like test different configurations
and know like you're basically freezing the state
of the factory at a given point.
So you can be like, okay, if we were to change
the factory definition, then like run all these tasks again,
we could see if things were better.
It also makes it so that like an agent can actually,
'cause these are all, it's all code,
like coding agents can actually update the factory
to make it better.
Does it make sense?
- Yeah, this episode is brought to you by OpenArt Arena,
the Global Leader Board for Creative Intelligence.
Every week new AI models launch
and everyone claims to be the best,
but best of what? OpenArt Arena is built
to answer the question that actually matters.
Which model is best for your specific job?
Instead of one overall winner,
OpenArt Arena ranks models across real creative tasks
from advertising and film to animation, product,
graphic design, editing, and lip sync,
covering both image and video.
And these rankings aren't based on hype,
they're judged by professionals, industry leaders,
and working creators through blind evaluations.
So judges never know which model produced which output.
That means you can see how models actually perform
when it comes to the creative work you're doing.
So stop guessing which model to use.
Explore rankings based on real creative work
and find the right model for your project
and save time and cost.
See the rankings at OpenArt Arena.
- Yeah, I just wanna like kind of sum up
where we are so far,
'cause I think there's so much rich stuff in here,
especially for engineering leaders and builders.
And then I know we're gonna get to some workflows
that are not engineering focused, which I'm excited about.
But a couple trends themes that I've seen as you talk
through this one work happens in public.
So I'm seeing this move towards,
instead of work being assigned in tickets
and then tickets being worked on on laptops,
what work is happening in public,
whether it's Slack or some other channel,
and then executed in the cloud.
So sort of anybody can interact with that.
And the factory does have this software development life cycle
kind of like definition to it.
But on top of that, it has this aggregate meta-analysis
that you're doing across all the behaviors,
which is giving, as we said,
giving the people the managers what they want,
which is, are we getting more efficient over time?
Is this factory actually effective?
How many humans, how many agents,
what's the interbalance between the two?
- You're right now.
And then what you're doing is you're also doing
these evals against key behaviors in your factory
that you want to correct.
And this is something that when I talk to engineering
organizations, I tell them all the time,
which is such a challenge when you're using something
locally, like, for example, cloud code,
is I say you need session level telemetry
because you need to be able to aggregate up the failures
within sessions across your engineering team.
And so some teams do this truly by, like,
sucking up every local coding session to S3
and running their evals kind of like on their own
in their own platform.
But I do believe that if you do not have session,
tool call, MCP call, test failure,
computer use level observability
into everything single coding session,
you're missing a lot of opportunity
to optimize efficiency, cost, model,
just like how your team uses the tools.
And so I think it's super important that people do this
and then what you've added is the sexual layer
of great than take those insights
and make the factory better,
however you define better.
- This is great summary.
- Oh, great, I did it.
- Professional podcast.
- There's, there's what,
let's just think of a sense of how you can continue
to like, you can go even further.
Once you have this data,
I think you're summary with awesome.
The other thing that you can do,
and like, I think Endorgs are going to do this
because this is how like,
there's gonna be nothing more important
than optimizing the way that you like,
build and ship software in the future.
And so one other thing that is really powerful,
if you set up everything you just said where it's like,
you have the, you have the tracking, you have the evals,
there's one further way that you can make it even more powerful,
which is adding the ability to basically take your own data
and replay those sessions with different configurations
to actually measure like, to your question earlier,
like, well, how would it have done
from a cost perspective or a quality perspective
if you'd done like different models?
And so for instance, again,
this is what we have built into work factories,
but there's lots of ways you can do this.
Like, the idea is you can,
just like there are these public benchmarks,
like sweet bench and terminal bench,
that are on like generic data,
you can recreate on your own data,
like from real past factory tasks,
how would things have gone?
Had you used a different model configuration, for instance.
So this is like a kind of Pareto chart,
which again, these things are on Twitter all the time,
but it's like, what's cool here is like,
if you take the factory approach
and are like really scientific around,
okay, I wanna know.
know like like I want to let's say you want to curate a bunch of front end tasks and then see like
can I be using jlm53 on those instead of using you know opus yes definitely should I be using
Gemini 37 flash you're going to take a quality hit and instead of like you get to choose your own
sort of trade-offs here and then you can feed this back into a model routing strategy where
you actually have evidence that okay on your own tasks for these types of like the best model
configuration for costing quality is is like whatever you find are you comparing that to what
actually shipped like how are you evaluating quality you know there's like seven ways to
skin a CSS like how do you actually decide you know this is better quality or not on front end
tasks for example yeah awesome question so the the way that we do it in more factories and
you're gonna probably you can do different things here is we use the exact same like scoring
infrastructure that I showed you earlier so for instance you know we over here we have all these
different scoring dimensions so you can if you have confidence in this scoring infrastructure
basically you're replaying tasks and trying to see if there was like how it affected these scores
so it's it's it's primarily LLM as a judge but you can also it's like you you could do this with
human judges you could do it algorithmically but the thing that we have built is LLM as a judge
got it super interesting okay so you've given me so many ideas on how to take the factory idea
further I'm curious just like take off our builder hat our CTO hat let's let's put on I you know
I was talking to you again before we came on it's like people love to see how non-technical people
can do technical things and how technical people can do non-technical things so show us some of your
other like I use cases that are maybe lots about evals and benchmarks and and MCPs and more about
you know begin a CTO yeah so we'll get like I think like zooming out this is cool with all these
charts I saw of a hard time making changes to Figma and like I I don't know if you're the same
way but when I go to try and change like a like a Figma thing I'm like it's like I have three
thumbs or something like I just I cannot figure out how to use it but I do know that it's like if you
like this is this is how we like do our slide decks for instance it's like we we are using Figma
slides you can make things that look very nice so one thing that I do now is like if I need to change
a slide deck I do it through the Figma MCP and the coding agent and so like just to kind of
show what that looks like we have this kind of semi-boring like slide here on cloud execution
so I'll show you how I would do this just so you get a sense so I'm going to do this I need
this in warp I use these warp as a coding agent this would also work in cloud cutter code accident
in the Figma MCP and so I'll just paste this in and then I'm going to talk to it which is another
thing I see some people around the voice train on the voice yeah we we give all credit to Hillary
Gridley she calls it the Yapper's API high-speed width way to talk to an LL
yeah so I'm gonna say like I'd like to make a new version of this slide use the Figma MCP to get
the context duplicate the existing slide rather than making changes directly to it let's have it be
so that the host box contains the runtime box let's let me see what else I got I got a switch back
here so we're gonna have host contains our host should contain sandbox I'll go back and edit this
let's make the context system something that's like kind of like you know cloud around these things
that feeds into them true CEO put a cloud on this one you're gonna see some bad design here
let's make the launch pad have a sort of like rocket type theme this is my this is not what
my cell stream wants by the way and let's make tracking and I want to you come up with some ideas
for how to make tracking better the overall idea here is to make this slide more visually appealing
than the simple you know five boxes and semantically show the relation of the boxes to each other
so I'll do this do you know what word I say to Figma uh I say semantically now what to Figma MCP
all really oh my god all the time semantic colors submit semantic semantic it is it is my orthogonal
that's really funny yeah I mean uh we'll see so and I'm using I'm using Groc a lot recently um
same I don't know what your model of choices I think Groc is pretty good from like the cost and speed
and quality quality trade off I can show you how you can do some other you know yeah why
why not while that's really why don't you show us something else yeah I'll show you another one
so I'm a big granola user as well and so again if I were really trying to improve this sales deck
another thing that I would want to make sure is that the sales deck is speaking to the things
that customers are actually talking about so I'm going to start a second task here using the granola
MCP where I'm like can you use the granola MCP to look back over my last four weeks of sales meetings
and try to build up a list of the top 10 frequently most asked questions in these meetings as they
pertain to warp software factories don't list any specific customer info in the summary anonymize it
because I'm doing a podcast so we'll we'll get this one cooking as well I can do one more if you
want yeah let's I mean let's queue them up again I love to see a AI pill CEO just open tab after
tab after tab and kick stuff off I mean this is stuff that this is stuff these are all by the way
of real things that I am constantly using AI for so the last one is like a task that I use AI
for is trying to rediscover potentially like cold leads that I might have talked to in the last
like you know six months or so you know our product has changed crazy amount I might want to
reproach them and so the thing that I I use for this or had in the past been using this actually
just started using instinct these instinct oh no I had a bad experience with instinct it about
experiencing this air so the the normal way I would do everything is still through my is still
through coding agent but I would let's let's I'll do one more prompt here can you use the
GOG CLI to look for emails and calendar events I've had in the past six months with potential
enterprise leads who might be useful for another outreach for warp factories you can learn about
warp factories at warp.dev/factories and make me a Google sheet with the info on them and share
the sheet link but don't print out any specific customer email or name in this thread what I like
about this at the metal level is like is this like your brain works where you're just like I need
to do the slide kick off the slide I need to like update some of our sales position kick that off
I need to like follow up with cold leads like is this yeah you panel is this a reflection of your
CEO brain because it's certainly a reflection of my brain yeah I mean if I were if I were in like
sales mode yeah go to market mode and like I am in that mode quite a bit now unfortunately more
more so than like straight up builder mode or end manager mode although I do all these different
things I'm constantly trying to find like what is the right positioning are we are we communicating
in these meetings in a way that lands and like it's so amazing to have a disability like
for me who can't design or draw at all or like to have this now super power and we can go let's
let's see how this figma is doing over here has it started to do it yet oh it's starting oh it's
got to do but it will it will do better but yeah if I was in sales mode like I'm trying to figure out
are we producing a right what are people asking about yeah and this is this is right like this is
from our actual sales calls and like this is that what I would say is the number one thing for the
factory like the product category where it is like bivers build and it's backed up by evidence and it
makes sense to me security comes up a lot workflow what's the workflow what's the measurement
how do we manage costs it's so cool and so it's like what I would
would do is like, you know, I want to then go make sure that our sales deck, our website,
all this stuff is speaking to the questions people have.
I might turn this into like a literal FAQ, like I think we have, you know, probably ability
to improve that or I was just trying to work this into our positioning, but I'm constantly
doing this and trying to understand if what we're building is the right thing and we're
positioning it the right way.
Oh, we've got a rocket ship. Sorry, we're back at for those that are not watching.
We got a rocket ship. We got a cloud of some sort. It's trying to, not very good yet.
It will get better.
The rocket ship's not terrible. It's not bad.
Um, Zach, this is, this is so fun. I want to go to quick lightning round questions and
then we'll get you back to, go back on the factory floor.
All right, first lightning round question. Everybody wants the factory. Everybody wants the
go-to-market intern that will happily go through your call transcripts and the design intern
that will happily make your ugly cloud slides as a CEO. But when you had the factory humming
and when you're doing, I think it was like over 2000 PRs in the last month. Like, how do
you keep things in the team from going as I say, like chaos reigns? Like, how do you keep
your arms around all that activity, all that work, all that context, all those tasks and
know at the highest level you're doing the things that matter?
It's a great question. Like, the kind of boring answer is like a lot of the stuff that
we've always done as software engineers still applies, which is like, you know, we're
dog fooding. So we're just like huge constant users of the thing that we're building and
making sure that the quality of the thing still works well. There's like cool knowledge
transfer that's happening, where it's, I don't know if you've seen this too, but there's
like a kind of power law for like how these tools are used in terms of like we have some
people on our team or super duper power users and other people are like kind of more at
the average. And so there's this, when you're working in this, what seems like a very chaotic
way in public with everyone slacking these factories all day, you do get a chance to sort
of see how like the really expert people are using it and that kind of up levels other
people on the team. And then like I said earlier, we're still doing, you know, it's not just
like chaos like everyone like, you know, slop stuff into the machine all the time. It's
like we still have a product development process that, you know, it's somewhat traditional
in a bunch of ways. So it's like we do a whole bunch of user interviews. We watch people
use the product. We do design jams before we start building to not not so much like figure
out exactly like how the UI should look, but just to make sure that we're tackling the
right user stories. Like because at the end of the day, like it doesn't matter how fast
you build software. It's like it's just also some user problem and solve it in a good way.
And so we were trying to figure out like, not lose the key parts of like the human input
here, but also, you know, use the fact that there's this like magic technology too that
can make you just go much faster. I love it. Amen. I could not say it better. Okay. Last
question. And then we will get you out of here when your AI is not doing what you want.
And also I am curious if this question is if you answer this question differently when
you're working with AI privately versus working in with it in Slack with it's not doing what
you want. What's your prompting strategy? Do you yell? I am lately. I will admit this
to the audience. I am doing a lot of like wire. Why are you like this? Why? Why? A lot
of questions. That's funny. I'm like a pretty level guy. I'd like to think if you asked
the people on my team, I don't know. I've never really yelling out my style. I get like
passive aggressive. That's kind of what I would say. I get like, I roll really like really.
Like this is what we're doing now. Like you built this whole thing that no one asked to
build. So I'll get like annoyed like in like subtle undertones, but I'm not a I'm not
a yellow. And so I don't know. And I also try to just keep the perspective that this is
just like a bonkers thing that I'm just like talking to this talking to this thing that
is doing this job that I've done for the last 20 years. And it's now it's like doing
it kind of better than me. And so I don't know. I'm still like, I mean like a little bit
of like a wonder phase with it. I don't know if you've ever seen that Louis C. K. Skit
where he he talks about how when internet like Wi-Fi first got on to airplanes. Stop
me if you know this. But it's like he's like he's riding on an airplane. It's like this
like 20 years ago. And there's Wi-Fi for the first time. And he's just like, this is incredible.
Like this is like amazing. And nobody's happy. And he tells the story about how like the
guy next to him is like trying to watch a YouTube video and like throwing his hands up like
screw this. Like I can't get a signal. And and Louis C. K. is like, like this is bonkers.
Like we're on an airplane. This thing is like going up to space. Give it a second. And
so I think it's so easy to lose perspective of like how wild this technology is like I'm
like I'm I'm generally like pretty probably like kind of more patient with it than most
I would say.
I love it. That's such a great perspective. Well Zach, this has been super fun. Where can
we find you? And how can we be helpful to you?
So you can find me personally like I'm on Twitter. I have Zach Lloyd tweets is my my Twitter.
You should obviously come check out a warp at Warp.dev. If you want to build software
factories, you know, I hardly even talked about it. But we have an extremely popular
agentic terminal that's open source that you should also come check out and use. It's
a great place to work with interactive coding agents. But yeah, this was awesome. I really
appreciate you having me on.
Yeah, thanks for joining how AI.
Thanks so much for watching. If you enjoyed this show, please like and subscribe here on YouTube
or even better, leave us a comment with your thoughts. You can also find this podcast
on Apple podcasts, Spotify or your favorite podcast app. Please consider leaving us a
rating and review, which will help others find the show. You can see all our episodes
and learn more about the show at howiipod.com. See you next time.
Podcast Summary
Key Points:
A software factory is a centralized, code-defined system that automates the entire software development lifecycle, from idea to code review and quality assurance.
Work in a factory is public and transparent, enabling real-time collaboration, visibility, and accountability across teams using tools like Slack and GitHub.
Human involvement remains critical, especially in code review and quality control, though automation significantly reduces bottlenecks and improves efficiency.
Self-improvement loops allow agents to analyze past failures, identify failure modes, and autonomously update factory configurations to improve performance over time.
Key metrics such as prompt interactions, PR velocity, and cost per task provide measurable insights into efficiency and automation maturity.
The factory enables real-time evaluation and scoring of agent outputs using LLM judges to detect issues like redundant tests or poor quality.
Centralized observability allows engineering leaders to track, analyze, and optimize agent behavior across all runs, enabling data-driven decisions.
The factory supports non-technical use cases—like sales strategy or design—by integrating with tools such as Figma and granola, demonstrating the versatility of AI agents beyond coding.
Summary:
A software factory is a comprehensive, code-defined automation system that streamlines software development by integrating multiple tools—like Slack, GitHub, and Figma—into a single, transparent workflow. Unlike traditional coding agents, it spans the full software development lifecycle, from idea to code review and QA, with public, trackable processes that enable team collaboration and oversight. A key insight is that human bottlenecks, especially in code review, remain significant despite automation, but the factory reduces friction by standardizing workflows and improving efficiency.
The system introduces self-improvement loops where agents analyze past failures and update factory configurations to prevent recurring issues. Centralized metrics—such as prompt interactions, PR velocity, and cost per task—provide engineering leaders with real-time visibility into performance and automation maturity. These insights are further enhanced through automated scoring, which evaluates agent outputs for quality, redundancy, and alignment with user needs.
The factory also enables non-technical use cases, such as refining sales decks or identifying customer questions from meeting transcripts, showing its broad applicability. Ultimately, the factory transforms development from a siloed, reactive process into a measurable, scientific, and scalable system where both human and AI inputs work in harmony to deliver better outcomes.
FAQs
A software factory is a comprehensive, code-defined system that automates the entire software development lifecycle, from idea to code to review and deployment, using agents and integrations across tools like Slack and GitHub.
While a coding agent helps write code locally, a software factory automates the full development lifecycle—including triage, issue creation, PR generation, QA, and verification—within a public, transparent workflow.
All PRs currently require human review, but the team has reduced bottlenecks by allowing prompters to also review the code they initiated, and uses risk scoring to auto-approve low-risk PRs for faster merging.
Factories track metrics like prompt interactions per PR, automation levels, PR cycle time, cost per PR, and agent performance to measure efficiency and identify bottlenecks.
Yes, factories use observer agents to analyze failed runs and automatically suggest updates to improve processes, such as reducing redundant tests or refining agent workflows.
Factories use automated verification, like computer use verification (e.g., video outputs), and scoring systems with LLM judges to evaluate test quality and detect redundant or flawed code.
Chat with AI
Loading...
Pro features
Go deeper with this episode
Unlock creator-grade tools that turn any transcript into show notes and subtitle files.