演进的旅程:从个体提效到团队落地的幻灭与重构
随着AI编码智能体的爆发,少数开发者获得了惊人的杠杆效应,引发了关于通用人工智能(Artificial General Intelligence: 具备人类同等或超越人类智能的AI系统)是否已经实现的广泛讨论。随后,许多企业试图强制全员推广,走向了极端消费Token的“Token Maxing”道路。然而,这导致了大量低质量代码(AI slop)的上线和频繁的线上故障。企业开始意识到Token的高昂成本与预算压力,资金不可能无限燃烧。在这个过程中,开发者对AI的态度在“失业恐惧”与“使用信心”两个维度上不断演变。从最初的高恐惧、低利用率,到看到早期采用者获得超常杠杆后的跃跃欲试,再到面对满屏垃圾代码时的怀疑与退缩。要让团队真正采纳AI,必须将开发者从恐惧和低效的使用状态,引导至无惧且高频的深度应用状态。这绝非简单地在代码库中添加一个 CLAUDE.md 或引入几个零散的工具技能就能解决的,实际的生产环境要复杂和混乱得多。
Original English Source
Today, I'm going to be talking about agents code bases and teams. Essentially, how do you get your team to actually ship together with agents? And like, I think for the longest time, like the one thing that's bugged me is like there's so much content about, you know, how do you set up your own code base to like work well with agents, you know, like what skills do you add? You know, this skill's better, that setup's better. But it all seems to break the moment you like actually try to use it with your team in your actual production setup. And like for individual repos, it kind of makes sense. But the moment you actually try to use it with your own like team setup, it tends to break. And like I think over the past few months, I've figured out how to make it work with a team of folks. So like I was leading a team of like 10 people over the last few months, and I think we found like a good solution, and I want to share that with you guys.But before we get into that, I just want to recap like, you know, what's been the journey that we've been on. Right? So we started, you know, code coding agents took off, and a few people got really, really good leverage, right? And I think all of us were kind of asking like, is this AGI? You know, like did we achieve it? Then, you know, companies took that and said, "Well, if one person can do so well, let's just get everyone and let's mandate it." And like token max, right? And that was clearly a a galaxy brain moment, right? And then the inevitable happened, you know, AI slop shipped, and there's a bunch of sev twos. I'm not going to name which companies, but you know, essentially you saw people kind of retracting. They said, "I don't think I don't think this is the best option here." And eventually model prices you know, climbed. We saw, you know, people figured out that tokens have to be paid for. Like you just can't token max your way through life. And budgets got bolted on. And you know, essentially money is being lit on fire, and the money has to come from somewhere.
So, given this journey, I want to actually like this is the enterprise journey, right? And what does that do for a single developer? And I think this is an important framing cuz it really talks about people as a part of a team, right? And I want to look at it from like two axes. So, there is the fear axis where people lie on the spectrum, right? They're coming It's like Is it coming from my job? Like am I going to be out of a job? Or is it like a really handy tool? And they're really, you know, they're not that fearful. Versus like the confidence they have in how much they're executing it. So, they can either use it a lot or they can use it, you know, not that much because they don't really know how to use it that well. Now, when we started, like people said, "Oh, you know, what is this? Is this the end? Like am I needed?" And fear was pretty high. Utilization was pretty low cuz people didn't really know how to use it. And then when a few people got outsized leverage, you saw, you know, early adopters basically people saw them and people said, "Okay, well, it looks like I'm still kind of needed if I figure out how to use this thing. So, let me actually try using it, right?" And then we saw, you know, mandates and token maxing, and people kind of got a little skeptical. Like confidence stayed the same, but people tried to use it a lot more, right? And then we realized, uh you know, there's a bunch of slop shipping, there's self-tours, and I, you know, it's like I'm not really that scared cuz, you know, it just slip it just ships slop. Like I'm going to still be needed. And they don't even know how to use it that well cuz like now the confidence is greater, right? And so you got to figure out how to get people from all the way wherever they are on the spectrum to where they're not fearful and they're actually using it a whole lot more. And this is the framing that I want to like I want everyone to keep in mind as they're actually trying to get a team to adopt good like AI usage and good AI patterns, right? And so, the The is, you know, what does it take? Step one, create a Cloud MD. Step two, add some skills. Is that it? Do we like solve it? You know, I think we all know like you guys are here because like clearly life's not that simple and like stuff's messy, right? And I think a few people might ask, you know, why doesn't this work? Like isn't that what like everyone does? And I want to like just talk about a few things you might see that actually indicate that yeah, like this isn't working.
诊断破损的Agent协作模式与领导力视角的重构
在团队日常开发中,有几个明显的危险信号表明当前的AI Agent配置已经失效:首先是看护Agent(Babysitting Agents: 开发者不得不时刻紧盯着Agent的每一步输出,频繁手动介入纠错),这完全违背了引入Agent的初衷。其次,团队成员开始频繁抱怨“今天的模型变傻了”。实际上,底层的模型并没有改变,而是任务的隐性复杂度增加了,这暴露了代码库本身没有为AI提供良好支撑的事实。此外,在处理并不复杂的任务时,Agent却无声无息地消耗了高达500K、750K甚至1M的上下文,触发了自动截断与压缩,这极大地浪费了资源与资金。如果任务调试需要漫长的会话和不间断的人工干预,或者代码库变成了产出糟糕代码的“Slop工厂”,就必须停下来反思。
要解决这些问题,首先要认识到将开发者与Agent的协作效率最大化,并非个体IC(独立贡献者)的职责,而是企业领导层必须投入的系统性工程。如果任由工程师个体各自为战,少数掌握窍门的“效率大神”每天能提交10个PR(Pull Request),而普通开发者每天只能提交1到2个。这会导致低产出的开发者不得不承担沉重的代码审查(Review)负担,导致他们不仅没有时间写代码,还要被迫阅读Agent生成的低质代码,从而对AI产生反感,形成恶性循环。因此,重构代码库以适配Agent,需要团队层面的共识与领导力的推动。
Original English Source
So, the first thing is if you're babysitting your agents, it's not the right setup, right? And you got to realize that. If you're seeing people in your team babysitting their agents, something's wrong. Is, you know, one of the things that I heard a lot was is, you know, insert whatever latest model there is being really dumb today. The model didn't change, right? The hardness may have changed underneath. But if it's really like that's that's acceptable to like small changes in the hardness, clearly your own code base isn't set up well. It's silently burning context and money. Like you don't realize it, you know, you go you blow through like 500k context, you might go to like 750k million and hit auto compact even though you're not doing like a really complicated task. Clearly something's wrong. You know, if you have long ass sessions needing constant intervention, there's still something that's wrong. If you're getting a constant slot factory, obviously know like things are things are not good. And if you're asking if you find yourself asking, you know, how are these other companies shipping so fast? Like how are model companies releasing models at like a month and a half two-month cadence? Clearly they they have something which we don't, right?And so, I guess everyone's kind of thinking so, you know, how do we solve this correctly? And so, I think the first thing to realize is we need to frame it correctly, right? It isn't really an IC's job. It's a job for leadership. It's a job for the company, right? Making engineers work well with their agents is truly the most impactful thing you could do as an organization. Because that's going to enable your engineers to ship faster and with confidence and avoid, you know, a lot of incidents. If we live in this figure it out for yourself paradigm, like people are going to get outsized productivity, some people aren't, and like the people who are generating like 10 PRs a day are going to like look like, you know, gods compared to people who are shipping like one to two. And the one to two PR people are actually going to get left with the review burden. And that's actually a really, really bad thing. Cuz now not only can they not ship, they're going to actually see bad code and then curse the agents and hence not be able to get on to the let's ship 10 PRs, right? And so it's really important to do this. If it's a problem facing the team, there's a few things you can do, right? The most impactful things that you can do to set up your code base to like make it work well require team buy-in. You can't just like if you want to change the way your code base is organized, you can't do that as an IC, right? And if you're treated as a leadership problem, then you can do things like this.
实操指南:共享Harness工程与三大核心原则
在具体的工程实践中,解决上述困境的核心在于Harness工程(Harness Engineering: 围绕特定代码库构建的、旨在为AI Agent提供精准上下文、测试套件和自动化反馈循环的支撑性架构体系)。这需要遵循以下三大原则:
- 智能上下文注入(Smart Prompt Injection):绝不能将整个代码库或海量文档一股脑地塞给Agent。相反,必须建立一种机制,在最合适的时间,将最精准的上下文注入给模型。代码库应该自带其结构蓝图和文档索引,当Agent检索到某段代码时,代码中的系统注解或注释能引导它去读取相关的详细Runbook,而不需要开发者人工干预。
- 闭环自愈系统(Self-healing Loop):由于AI生成的部分代码存在“幻觉”或不合规(Slop)是不可避免的,系统必须通过自动化流水线(CICD、自动化测试、静态扫描)来捕捉这些错误,并将错误日志自动反馈给Agent,使其能够自我修正,从而实现自愈。
- 持续迭代的共享机制:代码库的AI支撑架构不是一劳永逸的,随着业务发展和代码演进,必须持续优化。团队需要鼓励优秀的IC分享他们让Agent高效工作的最佳实践,并将其固化为团队共享的开发规范,而不是各自保留个人的“秘籍”。
在这一套思路下,团队开发了一项高价值技能:一键发布(Ship-it: 负责将完成的代码从开发态推进到准备好接受审查的PR态的自动化工作流)。这个技能会自动处理创建分支、运行测试、修复CI错误、撰写PR描述、以及处理代码合并时的全部评论。尽管这个自动化技能在运行时往往需要持续一个小时以上,但它彻底将人类工程师从无聊的审查等待中解放了出来。这种长周期的自动化技能能够极大地建立起团队对AI Agent的信任,让他们敢于“放手”去做其他更有创造性的工作。同时,团队还引入了“代码园丁”(Code Gardener)机制,每晚自动运行,对代码的组织结构进行体检,确保系统不会随着时间推移而退化。
Original English Source
So, the other thing this needs is harness engineering, right? Per code base. And I think there's a lot of content on this, so I just want to talk about a few principles, but I don't want to like make this talk about that because there's a lot of smart people. You're at AI engineer, right? Like this this conference is all about people telling you how to you best set up, you know, your code base to make things function well. So, I don't want to talk too much about this, but there's a few key principles here. Smart prompt injection is one of them. You want to treat your entire code base as one way to that so that you're able to like smartly prompt inject the model with just the right context at just the right time. Without you needing to do it. And that's the framing. You want to be able to say, "Okay, I've set it off on this task. It has like a map of how to find the things it needs at the time it needs it." If it's looking at some code and that code has, let's say some documentation, it the documentation needs to live in the comments. So, if it ever grabbed into that code, it reads the comment, goes to that file, finds all the information about it. That's just one example. The second is close the loop, right? You got to make a self-healing system cuz slop is inevitable. There is going to be some slop that's going to seep in. But, you need to have a pipeline and a way to close the loop to remove the slop, to detect it, and to be able to like self-heal the system. And then you need to iterate continuously. And I can't emphasize this enough. Like, you can't assume that you do this for a month and you're done. Like, things are going to change constantly underneath. So, you need to keep this as one of the things that you have to do as an organization. And the third most important thing is treat it like a human problem, guys. Like, this isn't It's not, you know, oh, it's this tool. Like, people will figure it out. Let's just mandate our way through life. Like, that's just not going to work. So, treat it like a human problem. Fear is real. Human emotions are real. We should recognize it.So, you know, enough gyan or as like it's more like the Hindi way to say like, you know, I'm enough prof like I'm kind of giving you sermons. But, like, how do you really do this, right? Like, these are all principles. What's the real playbook? So, here's what we did and here's like I'm not going to over like emphasize on this is the exact way to do it, but this is like roughly how we did it and you can take from it what you choose. The first thing is do the basics, right? You got to do them right. Progressive disclosure, I can't emphasize this enough, is really, really powerful, right? Find your best ICs and find what how they are making the code base work for them. Take those practices and pass them or guide. People can't live in their own practices. And this is really hard for engineers to do it. It's basically accepting that my setup is imperfect. And like engineers don't like to hear that. But you got to figure out a way to find those best practices and ship them across. Make sure that that's kind of a shared setup.
The second thing we did was there's one high value skill that we invested in. In our case it was this thing called ship it. What it did was the moment you're done with your code, it takes care of everything from code done to PR ready for review. Which means you got to open a PR, figure out your opinions, handle the handle all the comments, you know, handle all the PR descriptions, the merge comments, everything, right? It handles CI failures. It basically runs through these loops. And what this meant was often the skill was running for over an hour. And that scared people, but once you actually figure out once they saw the value, they get invested, right? Because it's one skill which tells them, "Okay, this AI thing can actually work for me. I don't need to constantly baby sit it. I can trust it."
The third thing and really important is to close the loop, right? So we wired issues and boards into the repo. Like we added CICD. We added gigantic reviews. We have like a code gardener that actually goes back and looks through a whole bunch of like every night it'll run and look at the code and check if something like not organized correctly. What does correct organization mean will depend on your code base. Get people invested and I can't emphasize this enough. You have to win over the skeptics. It's really easy to say like the skeptic is just someone who's scared. It's really hard to get them to buy in, but if you can get them to buy in, you know you're doing something right. You have to get them to be able to edit and play with the shared setup cuz that's the true way you know that they're actually invested, right? And this is where you got to it ensure you're iterating constantly. If people are like and this is the hardest thing for engineers again cuz you're basically saying I'm never going to like get to perfection in my setup. But you got to be okay with that. You have to do it and you You to treat it like, you know, X% of your IC time is probably going to be spent on it trading on this thing, which is not going to lead to like meaningful PRs like up front, but it's useful and it's worth it.
应对挑战与心智转变:走向“全意投入”与渐进式公开
在团队落地的过程中,有几个必然会遇到的挑战和对应的解决策略:
- 任务数量爆炸:在机制建立初期,由于没有做好合理的调度与过滤,Agent可能会在短短几周内疯狂生成四五百个琐碎的Issue,导致看板瘫痪。需要对Agent创建任务的权限和链路进行精细化配置,确保它们有序工作。
- 合并冲突(Merge Hell):当多个Agent和人类开发者同时并行产出大量PR时,代码冲突会呈指数级上升。这需要系统具备强大的自动化冲突调解和快速合入的机制。
- 原型开发与规范脱钩:为了避免实验性代码(如原型或临时脚本)污染主干代码库,应允许这些实验性任务在编写时退出主干规范限制,给它们提供隔离的沙盒环境,降低不必要的合规摩擦。
- 践行“渐进式公开”(Progressive Disclosure: 仅在需要时逐步向Agent暴露上下文,以降低认知负担和Token消耗):为了实现这一点,必须在架构上做减法。例如,将单份
skill.md文件的长度严格限制在100行以内,使其仅作为一个纤薄的索引,指向具体的详细规则文件。判断该机制是否生效的指标非常直接:在Agent启动任务时,观察它第一步的Token消耗量。如果瞬间消耗掉40K或50K以上的上下文,说明信息层级划分失败;理想的初始底噪应该控制在20K到25K tokens以内。
要让团队真正释放AI的全部潜力,开发者必须进行深刻的心智转变:从怀疑和“小步试探”,走向“全意投入”(Full Send: 像滑雪时必须顺应重力向下倾斜身体一样,彻底信任并把控制权交给系统)。滑雪时如果你出于本能恐惧而使用犁式刹车,往往更容易摔倒;你必须主动适应速度,才能学会流畅的转向。对待AI Agent也是同样的道理——我们必须接纳不完美,允许它在探索中犯错或摔倒,并依靠闭环系统自动恢复。只有真正信任并倾力投入建设这套“共享Harness”,团队才能真正感受到AGI时代的超凡生产力。
Original English Source
And, you know, I don't want to say this is perfect, right? Like we faced a ton of issues while doing this. And I'm just going to walk you through some of them. But it's an iteration loop, so you got to treat it like a piece of feedback. So, what are the problems we hit, right? There are too many issues. Like when we started like we blew up to like 4 or 500 issues uh without like, you know, just I think within like a couple of weeks, which is a crazy number for like a repo. And then, you know, there are so many like different agents all trying to create issues cuz they've not been wired correctly. There's a lack of agreement. People, you know, as soon as people saw like, "Oh, this isn't working perfectly or the way I expected it." There is super easy for them to say, "You know what? I'm just going to go back to babysitting my agent." You don't want that. You want to actually take their feedback and put it back into the skill and improve the skill. Agents are taking too long. This is actually one of those expectation setting things. It's good if agents take too long. That means you can actually go off and do other things and you have confidence that they're doing the right thing. At the end of the day like the moment we hit this reasoning paradigm, the longer the agent like thought, the better its output. You can treat like it's It's almost like a similar mindset for your entire code base and for your skills. There's going to be merge hell and like we just have to deal with it. We have to figure out a way to deal with this. There is going to be slop when you're going to write experiments. Treat it like it's own thing, right? What we said was okay, people are generating this code, but it's not relevant. It's not going to be shipped. It's just an It's a prototype. So, treat it like one. Get it to opt out of all the rigorous other standards you've got across your code base. And realize people vary on the spectrum, right? And depending on the day, depending on like what they're going through, they're going to vary on the spectrum. You have to be able to talk to them and figure out, "Hey, okay, why why are you facing this? Like is it if the model changed, the hardness changed, again you need to go revisit something. Figure that out.And I think the biggest like the easiest way to say this is instead of saying like the model is so dumb, like we have to ask how can I make it smarter or how can I edit like and not This is where I have like crossed out the my. It's not a personal setup. It's the shared setup that you have to invest in. And it's a mindset, right? You have to go full send. And like I want to end you with I want to end with this. Like, I learned skiing like a couple years back and the first like the hardest thing for me was you actually have to commit to it. If you're pizza braking, you're going to crash. Like, no matter what. You have to commit to the speed in order to actually get and feel like, okay, that's how I can turn and that's how I can truly ski. And so, I'm going to leave you with this. Like, just be okay with be okay with failing. You have to, you know, go full send and be okay with falling. It's fine. The point is to be able to recover from that and that will allow you to truly feel the AGI. Yeah, well, that's that's me and I'm happy to take any questions.
Yep. Uh so, the I'm going to repeat the question for the recording. Uh strategies that you found best for progressive disclosure. So, I think a couple things, right? Uh the first thing is even in your skill.md files, don't overload it. Like, we've kind of set a hard limit for like 100 lines in your skill.md cuz your skill is really a folder. So, that's step one. Make sure and I think I spoke about this during the talk, but when you have like some code that requires like a runbook, make sure the runbook is reflected in the comments so that if a code if somehow the code like the agent figures its way into like grapping into the code base and find that file, it knows I need to go look at this for like all the description of how this is relevant. Right? You have to organize and I think this is why I talk about harness engineering because your entire code base can be set up to encourage progressive disclosure. Don't overload your cloud MD or your agent's MD file into like one big thing. You want to make sure that like it's a thin index that can point through the right files and that's what the agent gets in its like first prompt cuz that's what gets loaded when it starts to work. So these are like some really powerful strategies that and the way you know this is working is when you give it a prompt. When you give it the first prompt, see what it's doing. Is it grabbing or does it know where to go? But how much context is it burning when it like immediately? So is it like I think like 20, 25K tokens get taken anyway, but like how much more is getting added? If you're coming to like 40K, 50K, like something's wrong. That's not really progressive disclosure. So you have to figure out these boundaries and then like based on this it's an iteration cycle. All right. Well, if there there aren't any other questions, feel free to find me. Happy to talk about harness engineering in general or like anything else, but yeah, thank you for listening.
📌 文中提及的人物和组织
公司/组织: Amazon AGI Lab