Hey everybody, my name is Eno. Uh really pumped to talk today about uh something that at Factory we care a lot about. uh when we started 2 and 1/2 years ago uh we said that our mission is to bring autonomy to software engineering. Um and that is like got a ton of loaded words in it. That sounds a little buzzwordy right now, but I think that the my goal is that you guys leave this like roughly 20 minutes uh with a bunch of insights that will apply to your organization uh and the teams that you build, the companies you advise, um and if you're building products in the space, uh insight into like sort of maybe how to think about building autonomous systems and also making your engineering org one that's able to use agents really successfully. Um, a sort of like plus of this is that ideally this applies to any tools you're using that involve AI. So it won't be specific to like our product or any of the other amazing tools out there.

Uh, you know, Andre Karpathy had a very welltimed tweet. Uh, so of course I'm going to mention it. Uh, you know, he he kind of talked about uh, this idea of software 2.0 coming from auto uh, the the ability to verify things, right? Um, this is something that's in sort of like the the mind of Silicon Valley right now as uh the most frontier models are built with post- training that involve lots of like verifiable tasks. Um, and really I think the most interesting thing here is the sort of frontier and boundary of what can be solved by AI systems is really just a uh sort of an input function of whether or not you can specify an objective and search through the space of possible uh solutions, right? And so uh we are used to building software uh purely via specification. We say like the algorithm does this and like input is x output is y. But if you sort of shift your mindset to thinking about automation via verification uh it is a little bit of a of of a difference in what is possible to build. Um and there is another great blog post by uh Jason where he talks about the asymmetry of verification. Uh this is like pretty intuitive to most people who know about like P versus NP. Uh it's like a a thing that a lot of people have talked about throughout the like history of computing and and software. But there are a ton of tasks that are much easier to verify than they are to solve. Um and and vice versa, but but the the most interesting sorts of uh easy to verify problems are ones where there's an objective truth. They're pretty quick to validate whether or not they're true. Uh they're scalable. So validating a bunch of these things maybe in parallel uh is easy. Um it's low noise so your chance of validating it is like really really high. Um and they have continuous sort of signals. Uh it's not just like a binary yes no but like maybe you're 30% 70% 100% accurate or correct.

Uh and you know the reason I bring both these things up is software development is highly verifiable. Right? This is like the frontier. It's why uh software development agents are the most advanced agents in the world right now. uh and there are so much uh there's so much work that has been put in uh over the last you know 20 to 30 years around the automated validation and verification of software that you build um testing right unit tests end to end tests QA tests right um the frontier of this is expanding there's tons of cool companies like browser base and computer use agents and all these things that are making it easier to validate uh really complex visual or front-end changes um docs right having like an open API spec for your codebase uh is something that can be automated. It's validated. Um I I I can go through and enumerate a bunch of these, but I actually think it is sort of a nice checklist for yourself, right? Do you have some automated validation for the format of your code? Uh do you have llinters? These things for professional software engineers are sort of like, yeah, of course we do. But I think you can go a step further, right? This is where that continuous validation component comes in. Um, do you have llinters that are so opinionated that a coding agent will always make code that is exactly at the level of what your senior engineers will produce? How do you do that? What does that even mean? Right? Do you have tests that will fail when AI slop has been introduced? Uh, and when highquality AI code is introduced, those tests pass, right? These additional layers of validators are things that most code bases actually lack because humans are pretty good at handling most of this stuff without the automated validation. Right? Your company may be at some test coverage rate that's like 50% or 60%. And that's good enough because humans will test manually. Um you may have a flaky build that every third build it sort of fails and everyone at your company secretly hates it but no one says anything, right? These are the sorts of things that we know are true about large code bases. And as you scale out to extremely large code bases, organizations with 44,000 plus engineers, right? Uh this starts to become a very accepted norm that the bar is sort of maybe at 50% or 60%. Um and the reality is is most software orgs can actually scale like that. uh it's sort of fine to be at that lower uh barrier, but when you start introducing AI agents into your software development life cycle, and I don't just mean in interactive coding, but really across the board, right? Uh review, documentation, testing, all this stuff. Um this breaks their capabilities. Most of you have probably only seen an AI agent that operates in a codebase that has uh a decent amount of validation.

Um I think a lot of the best companies in the world right now actually have introduced very rigorous validation criteria and it means that their ability to use agents is significantly greater than that your like average uh developer. Uh you know and and if you think about it this like traditional loop of understanding a problem, designing a solution to the problem, coding it out and then testing it uh sort of shifts if you have really rigorous validation. Uh it becomes a process of when you're using agents specifying the constraints by which you would like to be validated and what should be built. Uh generating solutions to that outcome verifying uh both with your automated validation as well as with your your own intuition. Um and then iteration where you continue to iterate on that loop. This move from sort of like traditional development to spec specificationdriven development is one that we're starting to see sort of bleed into all of the different tools. Different tools have spec mode. Droids have like our Droid is our coding agent have like specification mode, plan mode. Uh there are entire idees that orient you around this like specificationdriven flow. Um and if you combine these two things together, this is really how you build reliable and highquality solutions. So if you think about it, what is like the best decision for you to make as an organization? Is it spending 45 days comparing every single possible coding tool in the space and then determining that one tool is slightly better because it's 10% more accurate at Swebench or is it making changes to your organizational practices that enable all of these coding agents to succeed and then picking one that you're, you know, developers like or honestly letting people choose from the tons of amazing tools out there. And when you have these validation criteria, you can actually introduce way more complex AI workflows to your organization, right? Uh if you cannot automatically validate whether or not a uh a PR is like reasonably successful or has code that won't definitely break prod, uh you are not going to be parallelizing several like agents at once, right? you are not going to be decomposing a large-scale modernization project uh into a bunch of different subtasks like that is that is a very frontier style task to use AI for and if the single task execution right the simple I would like to get this done here's exactly how I'd like it to be done and here's how you should validate if that does not work nearly 100% of the time you can sort of forget successfully using these other things at scale in your company.

Um when you get into other tools like code review, right? Uh if you want a really highquality AI generated code review, you need documentation for your AI systems. Uh and yes, uh agents will get better at, you know, picking out, you know, whether or not to run lint or test. They will get better at finding solutions when you don't have explicit pointers. They'll get better at search, but they won't get better at just randomly creating this validation criteria out of thin air. Right? This is why we believe software developers, by the way, are going to continue to be heavily involved in the process of building software because your role starts to shift to curating the sort of environment and garden that your software is built from. You're setting the constraints. You're building these automations and introducing continued opinionatedness uh into the uh into these automations. Um, and you know, if your company doesn't have at least all of these, right? Then that means that there's a lot of work that you can do totally absent of a procurement cycle or buying one tool or trying out another one. Uh, and so plug is that we help organizations do this, right? I think that it's great to have tools that allow you to uh go in and assess this stuff. They have ROI analytics that let you interact. Um but I think that for most organizations uh there is actually like a very clear way to do this right you can go and analyze where are you across those eight different pillars of like automated validation do you have a llinter how good is the llinter do you have agents MD files an open standard that almost every single coding agent supports um you can improve uh and systematically enhance uh these different validation criteria uh and you can go through and say Well, we're seeing that coding agents are reliable enough for a senior developer to use, but our junior developers, if you have the tooling to to tell, by the way, like which developer is using what tools, you you you can ask questions like maybe our junior developers are actually totally unable to use these coding agents. And you'll learn that the reason why is not because they're like more incompetent or they don't know how to use the tool, but because there's these niche practices that you don't have automated validation for, right? And if you think about what what is the difference between a like Google or a meta and a uh a still large but like 2,000 person engineering or the difference is that a newrad with effectively zero context can go and ship a change to make YouTube's like boundary like slightly more round and it won't with some degree of confidence take down YouTube for like a billion users, right? And the reason that's possible is because of the insane amounts of validation that have to happen on that code for it to be shipped. The big difference that we now have is we have coding agents that can go and identify exactly where these gaps are and they can actually remediate those fixes. Right? So you can ask a coding agent, could you figure out where we're not being opinionated enough about our llinter. You can ask a coding agent to generate tests. We have an engineer named Alvin who I love this quote. He said a slop test is better than no test. Uh and I think that that's slightly controversial, but the thing that I would argue here is that just having something there, right, that it passes uh when changes are correct and somewhat accurately uh matches to the spec of what you want built, uh people will enhance it. They'll upgrade it and other agents will actually notice these tests. They will follow the patterns. So the more opinionated you get, the faster the cycle continues.

So I think that what you guys should be thinking about is what are the feedback loops in our organization that we are catering towards. If you have better agents, they will make the environment better which will make the agents better which will mean you have more time to make the environment better. And this is sort of the new DevX loop as well that organizations can invest in uh that will enhance all of the tools that you're procuring, right? So no matter whether it's a code review tool, a coding agent, etc., they will all benefit. Um and I would argue that it sort of shifts your mental model about what you're as a leader investing in when you're investing in your software work right now. The idea of uh you know opex as like the input to engineering projects like we are investing in we want more people in order to solve this problem. we need 10 more people. Um, I would I would argue that uh the other thing that you can now start investing in is this environment feedback loop that enables these additional people to be significantly more successful, right? And I think that that's the feedback loop that can actually take quite a lot of value because coding agents can just scale this out. So you know all of this is to say there's a lot that can be done outside of the like product itself uh to enable these systems and the best coding agents will actually take advantage of these validation loops right so if your coding agent isn't proactively seeking llinters tests etc then you know at the end of the day it's not going to be as good as one that will seek those validation criteria and in addition to that when organizations uh uh think about these sorts of things if you're the person who's able to say, "Here's my opinion. Here's how I want software to be built." It scales your capabilities out greater than ever before. Like one opinionated engineer can actually meaningfully change the velocity of the entire business if you take this to heart. Uh and you have a way to measure and systematically improve. Um so that's uh you know the the majority of uh what I came here to say. I think that the the the only thing that I'd leave you with uh is that when you think about where AI is going and like where we're at today, we are still really earn early in our journey of using software development agents. If you want a world where the moment a customer issue comes in, a bug is filed, that ticket is picked up, a coding agent executes on that, that feedback is presented to a developer, they click approve, that code is merged and deployed to production in a feedback loop that takes maybe an hour, 2 hours. That will be possible, right? We all are sort of skeptical about that fully autonomous flow. That is technically feasible today. The limiter is not the capability of the coding agent. The limit is your organization's validation criteria. So this is like an investment that made today will make your organization not 1.5x, not 2x, but that is where the real like 5x, 6x, 7x comes from. Um, and it's sort of a an easy thing to say and it's an unfortunate story because what that means is you have to invest in this. It's not something that like AI will just magically give to you. Uh it's a choice that you as an organization have. Uh and if you make it now, I can guarantee you that you will be in the top 1 5% of organizations in terms of edge velocity. Um and you will out compete everybody else in the field. So highly recommend investing in this sort of stuff and hopefully you found this helpful and have some lessons to take home. Thanks.