[Guest]: How do you scale the reviewing process? Because now that is blocking your senior engineers, it burns them out. One answer is: don't do any code reviews at all.

[Host]: This is Flywheel AI, engineering excellence, and we discussed the biggest bottleneck in software engineering right now: code reviews. What do you think of specification-driven development? Which of these guardrails gave you the most value? Does it matter which harness I use: Claude Code or Codex?

[Guest]: It matters immensely. You see what employees are doing is that they give people a handgun, which is AI, and then say, "don't blow up the internet, but use it."

[Host]: This is how the best engineers are solving the code review bottleneck right now. Here's the full episode: as we generate more code, this is already in the build phase where code review becomes a bottleneck. I've just come back from Google I/O. That was last week, and when the episode comes out, it will be a couple of weeks ago. And even there, Google and their people acknowledge that code reviews are a bottleneck, and we don't know how to solve it.

[Guest]: Yeah, Google was an interesting case because in 2025, they already reported that 50 percent of the code was AI generated, and then they are pushing for 75 percent. I always like to look at the big players to see what they do, because on one side, you see all of these crazy numbers, like efficiency gains. "We automated our entire deployment there with the AI and expanded it," for example, but very seldom do you actually see how they do that, right? So they report all of these gains. But then how do you translate that to your own work? That part is often missing for the big companies.

I think if you look a little bit about what they've published and what they write, you can kind of see that they have like an approach that prioritizes reviewing, more like it becomes most urgent because they've identified that now that you can generate code very cheaply, the bottleneck is moving somewhere else. Now some people, and this is another topic, but some people would argue that writing code was never the bottleneck, depends on how you see software engineering. But the truth of the matter is you generate a lot of smart code. So now you have more pressure on your reviewing, on your senior engineers. And I think what the big companies are doing right now is they are combating that with policies on reviews. In Amazon, for example, there have been some instances where there were outrageous revenue loss due to AI generated code, and then they put policies in place where they require some parts of the code or some parts of the systems to be reviewed by senior engineers before anything gets merged or deployed. So they are identifying that, you know, for some cases, AI generated code has to be scrutinized much stronger than in other cases.

So you can see that. And at the same time, you see other companies that say, "okay, we automate our entire PR review process," applying like this. How would you say... I like to think about it as a little bit like there's a horizontal way of scaling engineering and a vertical one. The horizontal one is where you automate processes you currently have, like PR review. You can say any PR that is created is automatically reviewed with Copilot, for example. And that's what a lot of companies do. But then they don't really talk about how that improves the quality, right? So they are automating the human pipeline that we have. What's in the vertical axis? The vertical axis is if you have a project and a small specialized team, you let them build their own tooling to make sure that the product ships and it's developed in the way that they intend to. So they are more involved in building custom-built environments for their coding agents and for the tools to deliver the software. Like, they don't have a blueprint that applies to their product automatically. They refine it. Yeah.

[Host]: And this is what I see as stacking of capabilities. So right, we have the model on one hand, then the agent kind of harnessed around a model specifically, and you're talking about an environment in which this agent harness operates. Well, how should we view them?

[Guest]: But maybe we should just go back one step. So we look at software engineering. I think it's a very established process by now, like if we think before the era of AI. Yeah, software development lifecycle, everybody knows it. And part of the reviews were done by humans, and it would work because code wasn't delivered faster than you could review it. That has changed now with AI. And so the question now becomes: well, how do we scale the reviewing process, right? Because now that is blocking your senior engineers, it burns them out. There are studies about that, where you know, cognitive debt is growing, people don't understand their own codebase because they just don't have the time to look at everything, or they don't want to look at the agent-generated code, etc.

And so the question is: how do you mitigate that? And one answer is: don't do any code reviews at all. Okay. But then the next question is: how do you do that? So I think one way to do it is to engineer the environment in which the agents operate with intention, so that you don't have to be the human in the loop to give the agent feedback on common things. And it starts with very simple things. So like I've just... let's call it automatic code formatting, or a security check. Lots of teams have integrated tools like SonarQube or something that gives you feedback, that then a human would take and fix the code with. And with the feedback from those systems, agents can do that automatically. So if you provide agents feedback on what they're doing wrong, and hopefully as close as possible to where the agent actually generates the code (which means on the developer's laptop, not necessarily in GitHub after your major commit or your PR), then you're suddenly in this world where you're trying to engineer feedback to the agents in a way that helps the agents to run for a long time without human intervention. So that's what I mean by building these types of environments. Yeah.

And I think it's very easy to get started with it. A lot of it will feel like, "but we've been doing this before," but now you're giving the agent that feedback, you're not taking it as a human anymore, right? And some of the models are surprisingly good at self-correcting themselves once you pinpoint to them what they did wrong and let the environment give that feedback. Yeah.

[Host]: Does the harness in this equation... does it matter which harness I use: GitHub Copilot, Claude Code, Codex?

[Guest]: Yeah, absolutely. What in my experience, the harness matters more than the model.

[Host]: Okay.

[Guest]: I mean, the harness is what provides the tools, what provides some type of prompting, provides the memory layer and all of these things, that then get submitted to the LLM to generate responses. For example, with the LLM, once a tool call is made, right? The harness provides the capability to execute the tool call for the LLM. It matters immensely. I made an experiment in one of my projects where I tried to implement a tool based on specifications and tests. So I had a full suite of specifications like specification-driven development, like many people like to do nowadays. Say like the idea is that if you specify your product exactly enough, you will be able to implement it exactly according to specifications, which is unfortunately not true. But an attempt that I made, and that failed. And then I thought, "okay, what if I use the ATDD idea and just generate all of the behavior tests up front, and then let that be the feedback to the agent who tries to follow my specifications?" And depending on the harness that I used, it worked or didn't work, even though I was using the top frontier model in both harnesses. Okay.

[Host]: Which harness gave you the best results?

[Guest]: Well, that keeps changing all the time. At that time, it was Claude Code. Now I would have said... I would say that it shifted to Codex for implementation work. But that's one of those things that's like a moving target, which is also one of the reasons why you can't really stop experimenting. There is this tendency for software engineers to try to systematize things and make it a process. And "okay, this is those, I know the rules." I don't think that works. Things are still moving too quickly. Okay, maybe I should walk that back a little bit. It works at some level. But if you define in your policy "we must only use Claude Code," who knows what will happen when the next release comes out? Yeah, it might behave completely differently because the models have different quirks and different personalities. Some are very good at instruction following, others are very good at filling in the gaps if you don't provide enough context as a human. So you need to be careful which model you use as well. Yeah.

[Host]: I want to zoom in on what you recommend for harnesses. But before we do, following your train of thought of specification-driven development and solving the code review process in that way, you mentioned the TDD approach worked better than the specification-driven development approach with a model following instructions in that way?

[Guest]: Well, I would say, specification-driven was my first attempt, yeah. And that didn't go well. It was a typical, "okay, I create the perfect prompt, and then the model does something different that I didn't intend." But when you give those feedbacks, in addition to telling the model where it went off track, that worked for me in that particular instance. And I would say, you know, those might be transferable to other projects, but it really depends on the project. Which again, brings us back to the need to be willing to experiment with what works.

[Host]: Yeah. And this feedback, just so I understand, this is in an automated way, right? So you have specs, then you have execution, and the feedback is not you telling what it did wrong. But what does that feedback look like?

[Guest]: Exactly. So when you're using things like CLI tools, you have the ability to feed back what is called, for example, a stop hook. So when the agent is done with its work, yeah, the harness triggers an event, which is a stop hook. And you can wire that up to a shell script that you run, for example. So you can automate running your test suite or your guardrails. I like to call them guardrails. And then you need to engineer your guardrails so they actually output like natural language text: "This is forbidden, do it in this and that way." So you base the feedback on the prompt that you would write as a human. So then you trigger that, and then that triggers the LLM to continue working on the thing. And you can pair these feedbacks that you give with things like rough loops, you know, Codex mechanism. And we have the Goal command, which really helps, so they will just keep running for longer until they've fixed the issue.

[Host]: Yeah, rough loops specifically allow you to run whatever you want to do in a loop, right? So it goes over and over. And this feedback mechanism would then be the input, to get it right and to correct it, go to something similar.

[Guest]: The idea is the same. The idea is the same.

[Host]: Yeah. And which of these guardrails that you've put in place gave you the most value? You mentioned linting already and some security controls. Do you have more examples?

[Guest]: Yeah, so I didn't know before, but I've since become a great fan of semantic grep (Semgrep). Because it allows you to put rules for things that it could catch in code, for example, certain types of code constructs. The example that I always give is: I don't want any default values in any of my methods in Python, for example. You know, you can set default values for parameters in Python methods. And this, in my experience, is one of the greatest sources of frustration when you have to review and debug the code later. So I just prevent that by putting a rule that detects these code patterns. And that then triggers an error: "You must not write it in that way, it's against policy." So semantic grep is really flexible. And whenever I interrogate the AI about the code that it wrote, instead of reviewing the code myself, some of these issues come up: "Why did you do it that way? It doesn't make any sense." They immediately trigger me: "Okay, let's add another guardrail rule for this particular type of pattern." So what then happens over time is you're shaping your environment, you're iterating towards it to be tighter and tighter, to be aligned with your preferences as a human, or how you think code should look like. And part of that is not only taste, but also you want to enforce that the AI generates code that is easy to read, not only for humans, even if you don't want to read it as a human, but also for AI. Code is context. That's why code quality matters. Yes, and if you don't watch it, the AI is going to confuse itself by the code.

[Host]: Yeah, the sustainability of codebases has always been that it needs to be simple and easy to change. And though that perspective hasn't changed with agents, we allow ourselves to generate more code, and the code that is out there might not be simple and might not be as easy to change. The building on top of that might be quite brittle.

[Guest]: Yes. And well, as humans, we also allow ourselves this little exception or say, "as long as this modularity is there, it doesn't matter if one of the modules is messy or something, as long as it is isolated, right?"

[Host]: Behind abstractions. Yes.

[Guest]: Yeah. And there's also something that I found with AI. If you keep things modular, that helps a lot. If you have very clear boundaries between modules, if you have well-defined interfaces that are not allowed to change, for example. And I would say that brings us to another type of guardrail, which is architectural constraints. So a lot of languages, they have architectural unit tests, and those unit tests execute extremely quickly. They only look at, you know, what dependencies exist between different modules of your codebase, so they can analyze it very quickly. And you can say, "okay, prevent, let's say, the UI from accessing the database directly, enforce that it always has to go through the business logic layer," or something like that. Because the AI also tends to make these weird interconnections between modules that a human would never do. Or if you just let the AI design your system and then draw your plot of your system diagram, you will see the weird things that emerge, and you encode them into additional architecture unit tests. Yeah, which I would count as another form of guardrail.

[Host]: Is there anything that you would say: "well, this is then still kind of part of the human responsibility that would not fit in automated guardrails?"

[Guest]: Yes. So something... I mean, if you look at software engineering and we're going back to the tradition of doing it, it always used to be: "Okay, what do we want to build? How do we build it? How do we keep it maintainable, right?" Keeping the codebase clean and simple means we can go fast for a long time. And how do we do that as well with architecture, right? We define the architecture up front, and then we started diving into the modules. How do we implement each of them? I think that should still stay the same. The models are not there yet where they can do that on their own. And my workflow now includes first understanding exactly what I want to build, exactly, so that there's no doubt, and then sketching out how this might look as a software system, right? Could be different services talking to each other, could be on a service level, your different modules talking to each other. Which functions do I need in the modules? So you specify pretty much the entire architecture, except the implementation. And once you have that, you can already encode that as rules. Yeah, you know. So this, I think that's extremely important to, also to combat... I think on social media, people already talked about the cognitive dissonance that comes from not reviewing code, where you don't understand your codebase anymore, and you're not able as a human to reason about your codebase anymore. That comes when you don't understand the architecture anymore and how components talk to each other within the software system. And yeah, you need to do everything to combat that as much as possible.

[Host]: Yeah, it's very hard for me to imagine where this is going to go. Because I agree with you, if I understand my software system, I feel like that's the skill that's really going to remain for engineers specifically. And I do believe that the implementation we're already automating. And I feel like with enough guardrails, we can hold code reviewing to where only the essence, potentially the behavior... I see people reviewing specifications. That will then also remain. But if I put myself in the shoes of an engineer, maybe someone that's listening who is in a bigger organization, you might not already know your system, then it's quite difficult to be like, "Okay, piece the puzzle in advance there." How much investigation do I need to do? That's all work that would happen up front now. And potentially before AI, you would do that kind of as you go, and you would iterate and you would improve. That way of working is now different.

[Guest]: Yeah, it's very, very, very difficult to be messy in your approach now, which is why some people might resist it. Because, you know, doing all the hard work up front, right? You're not just discovering, as you just said, as you go, you're doing all the hard work up front. You probably have to do a couple of iterations on your implementation to get a better idea of how it actually looks like. So that's not uncommon, but you can do that with live coding, yeah, right? And I think that is going to be the new normal for some time to come before things change again. But the interesting part is that, you know, I always ask people: "Well, how could you develop software if you didn't know what you wanted to build?" Right, that's also not possible. So you're doing the same work. Now, it's just that you're doing it up front, right? It might feel more intense, more involved, but that also means that this is becoming its own discipline. And when people think about, "you know, what should junior engineers do, now that AI can generate all the code," I would say, well, this is clearly something they can learn. That's an invaluable skill. Yeah.

[Host]: Right. I've seen the best software engineers kind of fall into two categories. The ones who are like you mentioned, someone had to build something, and I saw them stare at their screen (this was still before COVID while we worked at the office five days a week) and I saw them with kind of pencil and paper, and he was like writing stuff and drawing things out before even doing any hands-on. Yeah, and you know that's a very interesting skill. And then you see people who are like, "it's very hard for me to imagine, and I have to start typing," but then they're still fascinating and extremely effective in the way they operate. But it's completely like the inverse of what the first engineer would do.

[Guest]: Absolutely. And the cool thing is that if you allow yourself to do that, like to do the prototyping to go into this discovery before you put your specifications, not in stone, but before you kind of say, "okay, this is what we want to build," is that it's all incredibly rewarding. I really enjoy that because you really feel the speed that AI gives you when you iterate through ideas. I oftentimes find myself talking to an iPhone for one hour just to explore this idea, and maybe a tangent idea, and that maybe I should have gone on. But it teaches you so much about how you can think about systems, and it also elevates us as engineers because now you're not only thinking about, "okay, how do I implement this," but you also think about, "what do we actually need, right? What does the customer need?" So you can see that, I think, engineers get much more elevated. They start to think at that product level much more now because of that. And that's just going to continue versus before they were thinking about, "okay, but how do we write this in code?"

[Host]: How's it been for you? Because a lot of engineers I talk to, when they parallelize work, there's a lot of stuff going up here in terms of thinking, and execution power all of a sudden, and at the end of the day, they're exhausted.

[Guest]: Yeah. Well, they burn out because you're constantly exposed to stimuli and you're constantly context switching. Because, they... AI takes 20 minutes to respond to you. That's real, that's absolutely real. I talked to a developer last week who said he's burned out. He's just stressed, just on standby because he's concurrently reacting all the time. I think that's 100 percent real. As with anything, I think the way to combat that is discipline. Well, okay, the first step is you have to be aware of what you're doing. But you also learn how to interleave tasks, maybe on the same project but different things. So the work on the environment that guides the agent, that's its own project besides the actual project that you try to build. So, and I keep iterating between the two. So when I'm waiting for an agent to complete something, because I want to see what it does or how it did, I would just switch to something else. I might start another session with a different agent on the same codebase and start interrogating the agent about the codebase. And so I stay kind of in the same context, but I don't switch between projects as hard. Yeah. And if I don't do that, one thing that happens is that I keep having to type in the session: "Now please tell me, what did we do in the last half hour?" So it has to remind me what it did. And, it's weird sometimes, 100 percent.

[Host]: Yeah, I thought Claude Code does that automatically. Now if you're out of a session for too long, it gives you like a summary, so you don't have to ask anymore. Yeah, there... maybe some UX problem they're solving, I think.

And we already mentioned cognitive debt is going to be a very crucial topic that engineers who really care about this craft or about their work care about. And also, I spoke with Addy Osmani last week. He mentioned this, which I hadn't heard of: "cognitive surrender," where people are like, "the agent just takes the wheel." And if there's a problem, it's the agent's fault. And if it works, it's also the agent. But then, they really just let go of what they're responsible for. And I feel like that's quite risky in the way we are evolving. I think conversations like you and I are having are incredibly important to give people perspective and to also challenge what goes away and what remains as part of your accountability.

[Guest]: Absolutely. And the funny thing is, basically, what employers are doing is they give people a handgun, which is AI. And then say, "don't blow up the handgun, but use it," right? So there's a lot of use of AI, but all of that is going to be standardized. There are going to be policies on what you are allowed to do and what not with AI. And the example I gave with Amazon, right? They already made a policy distinction between your code that is more critical than other code, and how that should be reviewed. So we're just going to see the same thing, and it's just going to be another form of the process for software engineers, right? Where you have to define stuff, where if you just want to yolo it... "Okay, let's not touch the billing system, please." I think that's going to be normal pretty soon.

And also, a lot of the review work that we don't want to do, we can outsource to AI as well. So the idea of not doing any code reviews comes... I mean it's kind of like a loaded idea, right? But when you attempt to do that, a lot of realizations have to come with it, right? There are cheap wins like guardrails, right, that are deterministic, that execute cheaply and quickly. And then the conversation shifts towards architecture and specifications, and more validating specifications up front, rather than the code later. And likewise, even when you have to review the architecture later as a human, you can still use AI for that, right? You have a conversation with the AI, maybe have a specialized agent that you particularly set up to review for any violation of a specific type of architecture that you're trying to implement in your project. So you can also do an automatic scan over all of your code changes, and then use that as the point to start looking into it as a human, right? So you can still automate a lot. And it's really just about: "Okay, where should I look? Where don't I have to look?" Yeah, otherwise I don't see how we're going to scale the review part. The code generation is ten times faster or a hundred times faster. The pressure is going to be on all of the systems that come downstream from that, absolutely. And part of that is the review with the human in the loop. So we have to try to minimize that as much as we can.

[Host]: Yeah, I feel like the input we can likely scale faster, right? We can just execute on more ideas. The reviewing things in production with different tests... the reviewing process is really the challenge. Some of the things that you mentioned, I do see those as ingredients to solving this puzzle, that is the code reviewing process. Not having a human in the loop at all, if that's the big goal that we're working towards, even though it might not be feasible now with the models we have, I do see guardrails in there. I do see catering as much as you can. Behavior, functionality. I don't think there should be any reason to not write a test anymore because you can just generate the test according to the behavior that you want, right? If you solve a bug, if you make a test of that, that was already good practice, but there's no reason to not do that anymore nowadays. So that's a trend.

[Guest]: Yes. And what also people often forget is that generating a test is a small piece of code. The chances of the LLM not doing it right are much smaller than when you tell the LLM, "generate me microservices." Yeah, right? So that's also another point that I think helps when you write tests, because even if you generate them automatically (which is what I do), the chances of failure are relatively slim.

[Host]: I have seen and spoken to other teams, and they fully adopted specification-driven development as their workflow. They use one of the specification frameworks, and they're very happy with their approach. That's why I want to have them on the podcast. But they mentioned that they have not built the guardrails like you and I are discussing within their code review process. What they have done is their review process now very much revolves around the specification, and that is what is reviewed. And I do think they have some guardrails in place to make sure that whatever is in the specification is also translated into code in a certain way. For me, there's only kind of the risky part, which is the non-deterministic behavior that can be in there. But I think that's an interesting approach.

[Guest]: Yes, and actually good that you mentioned it, because guardrails do not also mean just a prompt, right? That's, I think, how guardrails started to be defined, if I'm not mistaken, but they can be deterministic as well. The specification-driven development approach, I really like that, but mostly because it gives a human clarity of what is being built. If you give that to an LLM and you observe what it does, you'll be surprised that after five minutes already, it starts to deviate from your intentions because you didn't specify something clearly enough or there was some room for interpretation anywhere. So you can iterate on your specification, trying to get it perfect, and then you switch the model, and it doesn't work anymore.

[Host]: So does this mean for you, your perspective, what do you think of specification-driven development? Is it something that works now, but it's going to be gone in the future? How do you see that?

[Guest]: Well, I think that is part of that discovery product process of what you're trying to build. I think it's extremely important to have that, that can also be used as automated tests later, like checking: "Did we actually do this? How did we do this? Did we do this in accordance with our coding principles?" Those are questions you can then give to an AI to investigate, so the specs will stay there. People like to go in-depth with specifications, like they treat it as if it was code. It's not, right? It's just a document of shared understanding. I would say that specifies part of their behavior, part of what is being built. I think a fine-grained behavior specification of whatever you're building is much more important. There was this idea of TDD where if you have all of your tests and your software gets deleted, but you still have your tests, you can rebuild your software. That was one of the ideas of TDD. And I've seen it work for the very first time when I had behavioral tests implemented that would give the agent feedback, and I had a specification as a prompt to start the implementation. That was the very first time in my life I had seen this actually work in my project.

[Host]: That's still quite incredible. I wonder if with existing codebases, right? A lot of this is in theory, which is why it's quite the challenge. But I wonder if, indeed, if you would throw out the code, if people have been doing TDD in that codebase for quite a while, if an AI indeed can then generate the code exactly how it is according to the behavior that was captured.

[Guest]: Yeah, if you have some time to burn, you can try that over the weekend.

[Host]: That seems like an interesting experiment. But what I have noticed is that also, the bigger companies previously, before AI agentic software engineering, I feel like they were trailblazing and a lot of smaller organizations were looking up to good practices, best practices, conventions and blog posts of what is out there. And what's happening right now, I feel like the playing field is quite equal, because everything is very quickly evolving, as you already mentioned. Over a few months, harnesses and their capabilities are pushing what is effective and all of a sudden what is now possible, which gives us a very interesting playing field to experiment with. How do you typically experiment to educate yourself? People have a constant trying to catch up, feeling behind type of thing.

[Guest]: But I think that's normal for everybody who's trying to keep up with things. So I like to use projects. I have a couple of different projects that are on a different scale. One project is purely agent-coded. I have never looked at the code, I've only prompted it to do stuff. I have one project that is a toy project, that's the one that I used for the TDD approach with a behavioral test and guardrails, where I test that type of approach. I have another project that consists of multiple microservices, which is much more complex, which is mostly event-driven. And I keep adding features to them using these different methodologies that I set up for this project, and just observe: what does the AI do? What does it do wrong? And more often than not, I see the AI do something weird, and then I make that immediately part of my default guardrails. So I kind of have a repository where we have default project setups for different languages, and that is preloaded with a certain type of guardrails that I found useful in the projects that I do. So the experimentation involves a lot of communicating with the LLM. I really found it useful to ask the LLM to explain itself. So if I give it a complex task, I ask: "Please tell me what your understanding is of what we are trying to do." A very simple question. So I'm trying to find out what the model understood of what I said versus what I actually said, and you can see how models interpret things differently. And that gives you an idea of, you know, personalities is too far, but what type of behavior does it exhibit, probably the better term to use. So I do that a lot.

I try out the latest tools that they give us. Like when sub-agents came, I tried that out. I concluded that I have zero visibility about what the agents are doing, like what are they communicating with, I have no introspection. So then what I would do is I try to add introspection to that. And I built my own assistant where I told the agent, "please spawn a sub-agent, but do it in a different terminal." So I can actually see what you sent to that different agent. And yeah, well, you'll see the weirdest things that the models tell each other, right? You can already see how the model started to deviate like at the first step when the handoff of the task happens. So trying to look under the hood as much as possible: what information gets exchanged, monitoring what the agents are doing when they talk to each other. I think those are incredibly important things for the orchestration itself. And I learned the most from looking at those types of things, of what the model is doing, how they talk to each other. So I think that is how I spend most of the time to refine my understanding of what the models do and also trying to apply what they give us in terms of tools.

[Host]: If someone is listening and they want to apply agentic software engineering to their own way of working, some piece of code that is already live in production, and likely the team has already adopted some practices or not at all, and they want to kickstart something, what is step one that you would recommend for an individual developer? And they could be within a team, I think that's the most profound case.

[Guest]: I would say the simplest is: start with guardrails, static checks. Or they should have, of course, a code formatter. You should have a linter. Try to write some Semgrep rules that enforce some of the best practices that you see in your codebase. You can ask AI, for example, "what are some anti-patterns that we use in the codebase?" and write a simple test that flags them. If you work in a team, have a team discussion. You know, if you get resistance, well, then try the next best thing. It's really difficult to convert people who are not buying into the ideas of trying to improve the codebase, and streamlining it with tests or with guardrails. But you can run all of these things on your local machine. You don't have to enforce everybody to use it, but you can use it for the code that you write yourself. You know, you pair it up with a smart script that checks whether you created that file or somebody else, and where you enforce your guardrail rules. And the key point is: you should measure, or at least get an impression of how it works with those guardrails that you put in place versus without them. Because once you see that they actually help you do more work and be less in the loop, you will not go back. I don't think it's... yeah, it becomes really convenient to not have to worry about and babysit the model all the time.

[Host]: I can see that, right? This term, "babysitting," I'm trying to get rid of in the conversations that I have with people, and I'm trying to figure out what the best way is. It could be indeed kind of experimenting and going into this cycle of continuous improvement. And it could be you catching behavior you want as guardrails for the agent to then pick up and do something with. And it can also be feedback that you get from your team, right? And you can create a pull request, then this feedback from your team, yes, it can also then catch things in more guardrails.

[Guest]: And that's also another idea. So what you can do is you can just say: "Hey, analyze my session logs on this project." It'll go into your home folder, and will start looking at all of your conversations. And you can ask if it sees any patterns where I had to repeatedly remind it of a certain thing. So you can examine your session logs to see where the model goes wrong, where you also have to correct it, and turn that into a static check, right?

[Host]: This feels like a product that I would want to have, yeah. Which is actually not a bad idea.

[Guest]: Yeah, well, actually it's a skill. You can just use a skill. It's easy, it takes like 15 minutes, and then you can use it absolutely.

[Host]: We mentioned different harnesses might have different results. Yeah, and in organizations, people just get one harness. And that's right, we've gone all-in on GitHub Copilot because of all the models, you know, but it's only one harness. And would you recommend anything to either those developers or those organizations?

[Guest]: That's a really difficult one. If you cannot choose your tools as a developer, I don't think any developer is really going to be happy with that. I think education would help in that case. You know, there are studies out there that compare harnesses. We have enough evidence to suggest that relying on one particular tool is not going to keep us at the edge. So there are lots of arguments to be made. I understand that organizations, you know, have this desire and tendency to just standardize things. But it's not like you select a product and then you run with it for five years. That's not how AI works. And that just needs understanding of what AI actually means.

[Host]: Yeah, I feel like that's missing, right? Because we are now beyond models specifically, and the harness might matter more than the model itself.

[Guest]: Well, there's another thing you can do as a solo developer: try to find out at what things your particular harness is good. Maybe it is at, you know, writing pull request documentation. Maybe it's really good at debugging. Try to find the use cases where you can still use it, and then you use it for that. Yeah, absolutely.

[Host]: For the people listening, I'm trying to do this new thing because I want your advice on what experiment you want people to run. If they have a week's time, and they can do one experiment as kind of their budget of time they have, what would you want them to do an experiment with?

[Guest]: Well, that depends on where you are in the organization, I'd say. But for developers, I mean, there are a couple of options, and it depends on what you want. Do you want to get better at agentic engineering, like working with AI? Do you want to get better at putting in your guardrails? Do you want to get better at understanding how the models behave? So that's the first question. But whatever you want, there's an infinity of many different options of what to do. I would recommend: if you want to get started with guardrails, start looking at semantic grep (Semgrep). Let's try to encode some of your human feedback that you would give in a PR as a rule. And the example that I gave earlier with no default values in method parameters could be one example. Another one could be: "let's never swallow any errors, and any error must always be propagated," something like that. Just start experimenting with that and see how your agent reacts to that environment. So setting up, I think, the guardrails so that you can start to improve them and iterate on them, I think that's very worthwhile. And you will see, I would argue that you will see wins very quickly. Even though you might think, "but we had those static code checks first," now you're doing it in a way that you encode your human preference as rules right now. Yeah.

[Host]: So the customer for your particular project, clearing. And thanks so much for going on. This was really valued. This is really good. Thank you.

[Guest]: Thank you.

[Host]: If you're still with us, let me know in the comment section what you thought of this episode. And we'll see you in the next one.