破局技术债：医疗索赔巨头 Wisedocs 的重构抉择
在 2025 年，Wisedocs（Wisedocs: 专注于复杂医疗索赔文件处理的 AI 科技公司）面临着业务快速增长带来的严峻技术瓶颈。由于公司需要处理高达 10,000 页以上、甚至比视频文件体积还要庞大的复杂医疗 claims PDF 文件，底层的 AI 管道（AI Pipeline）承受着极大的压力。当时，系统面临三个核心痛点：首先，系统吞吐量低下，响应速度无法满足用户激增的需求；其次，由 10 多个独立代码仓库组成的遗留系统极其复杂，导致 AI 工作流更新困难；最后，由于技术债高企，团队开发体验极差，无人愿意涉足这些遗留代码。
面对这些挑战，团队做出了一个关键决策：投入 6 个月时间，将这 10 多个离散的存储库彻底合并为一个单体仓库（Monorepo: 将多个项目的代码集中管理于单个仓库的工程模式），并对整体 AI 管道进行深度重构。然而，在 AI 工具飞速发展的今天，这种耗时半年的重构决策是否合理？如果我们把技术债（Technical Debt: 软件开发中为了快速交付而采用易于实现但后期维护成本高的设计方案）比作金融债务，它会以神秘且不可预测的方式进行复利累积。因此，评估技术债的 ROI（Return on Investment: 投资回报率）变得至关重要。如果我们引入了额外的复杂度而没有带来成比例的业务增长，技术债的累积速度将轻易吞没其产生的收益。
Original English Source
It's not just my AI pipeline that's on fire, but also my PowerPoint. So, it's 2025. Uh, we're scaling as a business and things are going poorly. We're adding too many customers. We're not getting the throughput we need and we need to improve our underlying technology. And there's three main issues that we're facing. The first one is that we're too slow to meet customer demand. The second one is that this AI pipeline that we've built is too complicated to update. And the third one is because it's a legacy codebase or actually more than 10 repos, nobody actually wants to touch the code. It's not a fun experience. So we made this decision to refactor over the course of 6 months. And the real question for this talk today was this the right move to do. So I'll spend this time answering this question, but let's start off with the use case. So the company I work at, Wise Docs, processes complex medical claims, which are PDFs that are more than 10,000 pages in size. Some of these files are bigger than video files. So it's a pretty complex application and because of this, it's actually non-trivial to scale the different parts. So we're going to talk about the pipeline today, which has a number of ML models. So I'll divide this talk into a number of chapters. We'll start off with the first one, which is the concept of tech debt. So I think we all have this feeling universally if we've been developers for a while that we all write bad code. The question is do we do this intentionally or not? If I look back to some of the earliest code I used to write uh it was bad. This was more than 15 years ago. I tried to print an image of this character from a video game and I didn't understand that you can't uh system.out.println in Java to render something on the screen. So hopefully I've come uh further from that point in time. But there's these moments where we all know that we've written bad code before. Now, if we think about technical debt as financial debt, it compounds in mysterious and sometimes unexpected ways. But you should think about it in a rigorous format as well. For us to achieve some kind of ROI by taking on technical debt such as building a feature or getting new customers, we want to make sure that the ROI makes sense. If we introduce additional complexity into our codebase, we can very quickly outrun the ROI we've generated.AI 工具的进化演变：重构周期的指数级缩短
在技术生命周期快速更迭的背景下，近年来以 Anthropic（Anthropic: 研发 Claude 系列模型的领先 AI 实验室，曾发表 Spotify 和 Stripe 的重构案例研究）为代表的机构展示了 AI 在提升开发速度和辅助重构方面的巨大潜力。在 Wisedocs 重构之初，团队耗费了两个月时间来评估 5 个开源的 AI 管道编排工具。当时，由于诸如谷歌和 OpenAI 的深度研究搜索功能（Deep Research）尚未面世，所有的调研、对比和要求收集都必须依赖人工，并整理到 Confluence 文档中。
但在一年半后的今天，得益于 AI 技术的爆发，类似的工作流程已经可以完全实现智能体化工作流（Agentic Workflow: 利用 AI Agent 自动执行任务、调用工具和进行多步骤推理的自治工作流）。通过部署专注于不同评估指标的子智能体（Sub-agents），团队可以自动进行深度调研、编写 POC 并完成横向对比，将这一评估阶段的效率提升 90% 以上。
同时，代码编写和生成的验证效率也发生了根本性改变。在重构的早期实验中，演讲者曾尝试使用 Cursor 配合早期模型进行 Temporal 工作流的重构，整个过程经历了长达 3 小时的反复对话微调，且模型犯下了 10 个致命的逻辑错误，仍需要大量人工介入与代码删改。而当使用最新的前沿模型（如 Claude 3.5 Sonnet 与 OpenAI o3 等）重新运行相同的基准测试时，新模型在拥有更完善的验证工具链和 Agent 规划能力下，不仅能够一键生成（One-shot），而且只需一次微调迭代即可完全解决问题。这种高效的自动化让相同的重构任务在当前环境下只需原本 1/5 的时间即可完成。
Original English Source
Now, with AI engineering, you've probably seen a number of different stories that have come out to showcase the progress that's been made. These are two case studies from Anthropic, one from Spotify and the other from Stripe, talking about the immense progress that they've made both in shipping velocity and also the ability to refactor code. So, at this point in time, writing code or making changes is something that teams are doing faster and faster. Now, I'll pause here. Who here thinks that products have gotten better in the past 20 years? Technical products want to raise their hand. I hope everybody, right? Phones are pretty cool. About five years, a past year. Okay, so the challenge is that we're going to fast and faster through the technology life cycle, but we've lost something. The the product focused on customers in some way has degraded. The maintainability of the code and the reliability has degraded. You can see some of the up times here from two leading companies. I've blurred out their names for for it. It doesn't actually matter who they are, but we are below a 39 or even 49 reliability. So even though we're shipping faster and faster, the code quality and the product quality has not necessarily gone up. So let's talk about the refactor that we did. So we started this refactor with actual code implementation in April and did some pre-work earlier. So I'll go through five different tasks that we did and share some of the the findings that we had before and after especially with as new models have come out. So we spent around two months evaluating orchestrators for our AI pipeline. We looked at five open source projects and we wanted to benchmark and see how effective they were for our use case and we started this off before deep research came out as part of Google and open AI so that web search capability to do a comprehensive analysis was still not there. Now, after we actually gathered these requirements, we built out proof of concepts with a team of three to make sure that we actually got the right results. Now, I'm pretty confident we could do this 90% faster now with the tooling that we have. Before, we would manually go through, use a little bit of AI, but put everything into a Confluence dock and we'd evaluate across 17 different criteria that we came up with. Nowadays, we could build a much more agentic workflow to do that. starting off with deep research uh making sure that we match that against the problem statements that we have creating sub agents for each of these criteria and uh products and then finally building PC's and evaluating so things have changed in the past year and a half where we could actually go much much faster but we still have to maintain that same set of quality because it's very easy to undergo AI psychosis where you look at a deep research report that's 20 pages long and you say wow this looks good and then those features don't actually exist in the product and you've set yourself back. Now after we've done the initial uh orchestration research and model serving research, we went into actually committing code. This is just a example of what happened when we were experimenting. I was doing some initial research with temporal and committed some activities in workflow code to make sure that we can actually replicate what we have in the legacy codebase. So then I did what we wanted to do over a number of iterations and at the time gave it to o3 to actually try to implement this code and it did it much faster than I would be. This refactor took uh 3 hours of back and forth chatting within cursor but it made 10 major mistakes. So at the time when we were going through this refactor coding was getting better and better but it still hadn't reached the point of where it is now. And it was still a very manual process where you had to intervene and actually guide the model and manually edit or delete code. Now I reran these benchmarks on some some modern models. So we have sonnet 4.6 and opus 4.8 and things are much faster. Uh sonnet 4.6 with one additional iteration was able to solve the task and with opus it was basically able to oneshot this problem. So models are getting significantly better along with harnesses. And the interesting part here as well is that the way that the models interacted has changed substantially as well. Before with o3 there weren't substantial tool calls on certain categories. Uh and then as we moved into sonnet 4.6 and opus we see now that in modern harnesses we get sub agents we get some of those plan calls we get uh different shell commands and we get different verifications. And overall this process even though the model execution was a little bit more expensive it was a lot less manual. So we could actually accomplish a lot more. So if I was rebuilding the same task that I had for for this refactor, it would take around 1/5 of the time to accomplish which is pretty good progress. So I think all of us realize a scenario that models are substantially better now than they were before. Now this is really important because it shapes the way we think about the software development life cycle. We think about 2025 and the types of work that we were doing. We were making some small changes. We would give specific code snippets to models. We were just starting to get into this agentic framework uh of the type of work we can do. And now if we provide a a well-constructed spec to a model, it could generally execute it at a very very high uh capability level. And we can see this both in anecdotal experiences as well as some of the thought leadership that has been coming out of the big labs that this image is one from anthropic.高置信度与认知界限：理性构建 AI 开发的心智模型
在将开发任务交付给 AI 智能体时，我们必须建立一个理性的心智模型（Mental Model: 帮助人们理解系统如何运作并预测其行为的内部认知表征）。在开发社区中，常常有人抱怨：“我在深夜十一点或者下午五点启动了一个智能体任务，等我回来时才发现，因为提示词、开发规划或原始需求有关键遗漏，智能体一直在做无用功。”
为了避免这种令人沮丧的“毁灭循环”，我们需要量化分析智能体在不同任务时长下的可靠性。
- 传统的评估图表通常基于 50% 的任务成功率 来定义智能体的能力边界。
- 然而，在实际的工业级软件工程中，我们必须以 80%、甚至 90% 或 99% 的置信度 为标准来衡量 AI 的表现。因为没有人愿意在只有 50% 成功概率的事情上浪费大量算力和个人的注意力周期。
根据业界对前沿模型能力趋势的追踪，目前即使是最顶尖的模型，在执行时长超过 4 小时的复杂任务时，其成功率也会发生断崖式下跌。而在 15 分钟以内、甚至 15 秒以内的短生命周期任务中，模型的执行成功率表现最为稳定。这意味着，当前的软件开发生命周期仍需要人类在宏观层面提供清晰、无歧义的 Spec（Specification: 软件设计规格说明书），并通过合理的任务拆解，将 AI 控制在最高效、高置信度的执行区间内。
Original English Source
Let me ask the group a question. Who here has kicked off an agent and realized that either the prompt, the plan, or the requirements were incomplete or missing? A lot of people. Yeah, it's it's very frustrating, right? You you're like, "Okay, I'm ready to go. It's 11:00 p.m. or 5:00 p.m. I'm going to set off an agent and then come back." And then you realize there is a there's a critical flaw. Now, the reason I bring this up is it's very important to have a good mental model in understanding how accurate models can be in accomplishing tasks. Who here has seen this meter graph before? like decent number of people. So this is pretty common on actually mapping how much time models can complete tasks of certain categories for so the idea being that as models get better and better they can do longer running tasks. Now typically this graph is shared with the 50% accuracy rate but I think it's much better to actually look at the 80% accuracy rate or higher. And you can see there you can still see a similar exponential trend but we're no longer claiming that models can accomplish tasks that would take a human 18 plus hours. Now, I actually think it's much better to measure the accuracy at 90 or 99% because this is where the mental model is most efficient. You construct a plan, you create a spec, [snorts] you hand it off to an agent, and you're pretty sure that it'll get things done, right? You don't want to be creating a plan or a spec and then have a 50/50 chance of coming back and knowing that you wasted compute and your attention span. Now, if you're kicking off a process that is going to take an hour and it has a 50% chance of completing, there's a very high chance you just wasted that hour and you could have been doing something different. Now, if we think about a broader evaluation, so Meter does have some more information about their frontier models. So, this is one for for Mythos preview that they did roughly a month ago. And you can see here that generally the success rate starts to decline significantly at that 4hour mark. But even before then at the 15 second mark or even before the 15 minute mark there are certain tasks that mythos uh in in all its glory cannot complete uh effectively and consistently. So we're making rapid progress in in the AI model space but we're still not there where you can just kick off an agent and have something be completed reliably. So again this is really important for your software engineering teams and for you as an IC to understand what is your mental model and how are you going to contribute to that. Now I think what's really important is I think you've been hearing this throughout this conference is that there are a number of different frameworks and primitives that you need to have implemented in order to have good agentic development and this is no different from from what we found as we were continuing to mature as an organization and going through our refactor. These are the things that made sure that we can implement uh the solutions effectively and not waste our time just running doom loops with models.单体仓库的生产力回馈：Wisedocs 重构的最终成效
通过将原本杂乱分布在 10 多个不同仓库的代码彻底整合进单体仓库，Wisedocs 获得了惊人的生产力跃升。在重构启动后的前 6 个月内，随着系统功能对齐，团队的交付速度曲线呈现出极陡峭的上升趋势，且在后续新功能的持续迭代中并未放缓。
重构对团队效能和开发体验带来了以下直接改变：
- 更低的代码贡献门槛：在重构前，只有极少数对特定仓库熟悉的专家能够进行修改；重构后，由于统一了数据 Schema、API 调用方式和技术栈模式，几乎公司内的每一位开发者（甚至是其他业务线的工程师）都能够无障碍地向这个单体仓库提交 Commit。
- 开发环境与测试闭环：单体仓库极大地简化了沙箱测试环境（Sandbox）的搭建与端到端测试（E2E Testing）的验证流程。相比于以往需要耗费大量时间去分别克隆、配置 10 多个仓库，如今的 AI 智能体在单一目录树下能够更轻松地理解系统全貌，并高效完成部署和质量验证。
- 业务价值的兑现：尽管在重构过程中全部采用了人工 PR 评审来确保代码质量和团队上下文同步，但这笔重构的“时间投资”完全匹配了公司的业务预期。AI 管道的运行成本和响应延迟大幅降低，过去需要耗时数月才能交付的复杂业务功能，现在在不到一周的时间内即可顺利发布上线。