多人软件工厂：应对多智能体并发交付的挑战与架构方案

For example, um we have a lot more merges and they have to make sure that they're not conflicting and if we're unlucky, the agents also spend time of going rounds around us. Uh you know, one agent changing something then the other agent changing it back. These conflicts are real when you turn on a multiplayer factory. And one of the most noticeable things is we start to wait more for releases because when we merge a change all the other PRs need to rebase and they typically rerun their tests uh and you'll have to wait and at the same time that means actually your CI/CD system is becoming a lot lot slower because of that multi- thing and also more costly but that's a side effect.

Now the PRQ is kind of building up not just with features. There's also the agent incidents that we have to solve as I mentioned and then there's factory maintenance on their own like the improvement of the skills the improvement of the loops that we've been using all more more and more work that needs to get approved by us. So why can't we flip this from being this concurrency and it being an exception to becoming the routine? And this is what I call conc concurrent delivery and going from continuous delivery to concurrent delivery and design for people to work and agents to work together on the same codebase.

And then once uh you start planning and you start actually executing the plan uh much like you would do on your uh scrum board, you would say I'm going to take care of this and the agent writes and says it claims the ticket. It says I want to do that ticket. So no other agent can actually step in at the same time. And it also announces which files and which changes it's going to make. So this is something that is helpful for other agents to see that they're not starting to overlap during the planning and the execution.

Now to make sure that they can keep executing um a lot of people use work trees to kind of separate them. When you kind of think about this broader there is a lot more splitting that you could do. You basically are phrasing the world of an agent. You give them their own kind of machine. You give their their own phrase of the context. to give them their own version of the API that's being locked and even the database gets their snapshot their test data and even outside world like digital twins and uh kind of making sure that everything's mocked and that they can do these things without having to deal with any impact of the outside world.

Now on a conflict you could rerun it. Um and within get you're merging today and you say okay there's a conflict. What do I do? You rerun and see if it merges on the rerun which was typically our PR uh change or you could just kind of make sure that the agents do that for you instead. So they can see whether they can merge instead of just by code lines. they can merge by function and it becomes smarter and then the agent maybe can fix [clears throat] the merge so you don't have to do that again and ultimately it can even start seeing changes coming so hey this agent is probably going to change this I'm going to see this coming and I'm already kind of adapting my code uh at the same time so this is thinking in concurrency instead of being kind of serial in the changes

now one of the things you can see is that you agents have been optimized to merge stuff, right? Like with the merging of code, this is clear. But what if two changes land at the same time? In this case, you're calculating and changing the function of calculating the the total for an invoice and say it's not just in dollars, but it in dollar cents. Now, at the same time, there is another PR that changes the fact of reporting out. Now two mergers that consistently could merge with the test succeeding but at the same time it was a different intent. The report don't want to have it in euro cents or in dollar cents. Right? So they're not conflicting in code but they're conflicting on intent. So we could have agents actually look at the intent whether they're still consistent across the multiple mergers and warn us or change the PR accordingly.

Now, if you want to kind start merging, um the PRs are able to merge, they're good to merge. Um but what if one of those mages uh merges doesn't uh merge well, right? Then you'll have typically rebase all the others, they'll have to do rework and so on. Now, there's a better way of dealing with these merge trains is by looking uh in a concept called speculative merge trains. And it looks at all the PRs coming in and it sees which ones are compatible with each other and it kind of already checks them in pairs. And then it says like when that is landing and this one isn't landing, but you could still do the others, it goes faster because all the others that can go through already go through in the merge cube.

Uh these are just some examples of kind of new strategies of dealing with merges and conflicts and reviews and better planning. I think in general there's a new kind of infra layer coming and you see that almost from the vendors you know it's a new developer infrastructure mergers are going to be handled differently there's now an analytics there's now a better planning there's now a better tool for anything and that's kind of what we're building up uh right now a lot of the factories are just glorified loops as we would do with the SCD but the concurrency will only increase

Now one of the nice features is and uh it's been kind of interesting to see that agents can join your chat and this is a different way of coding. It is not you prompting but it's you kind of pulling an agent into a conversation. Uh and that's really powerful because the agent is picking up clues not just from you from your team members at the same time. And what it's actually doing it is refining the ticket from all the input. And that is really powerful because you actually not only had like adversarial review by agents, but you also get multiple input by humans on the same ticket while during the conversation.

And what it's really good at is prompting you with questions instead of having to ask everything over and over again. So one thing that helpful is that it clarifies what is the test, what should it do, what are the requests uh requirements that it needs to do. And the ticket actually becomes almost like a plan the plan mode in your coding agent but now as a share feature within your board um of looking at all the requirements.

Now what's even more powerful is that it doesn't only have to be engineers. Think of this as in the past sales marketing support they were kind of filing tickets to the engineering that would get somewhere on a board and they all have to wait until an engineering comes that could have taken weeks. Right now what we see in our factory at Tesla is that people are just sharing and doing a chat within kind of the uh organization and the agent is there. Now they don't need to build um the uh uh the code all the time and before [clears throat] they were maybe doing vibe coding but one of the problems with the vibe coding is it didn't fit according to engineering practices. Now with a multiplayer factory people are using the same pipeline the same knowledge the same skills the same loops to actually already fit into that flow of engineering itself. So they can experiment, they can refine the feature and they can also even build the feature where before we would wait they can just kickstart it and engineering refused that. That's a very powerful flow of getting everybody on board and being in the flow of that one multiplayer factory.

Now what we really want to do is reduce kind of the choices uh that it has to kind of make and what is really powerful is that it represents actually questions to us for decisions not just what should I do and open questions you know you see that in coding agents as well it gives you a couple of options that you can choose for and then it goes off this is really powerful and it is less trained on like having to tell the agent all the time what to do by presenting options there as well. Of course, you can override, but this is really a powerful multiplayer thing.

Now, every decision that people make actually goes onto a ledger. This is like your context locally maybe that you have in your coding agent. But if you are in a multiplayer factory, this is shared somewhere on a decision log and that can be used either by agents, it could be used by humans to find and you can trace things back like who made what decision uh and when that context is being used in the next session, you can also query that as well.

Now we want that to be small because big changes uh they are not very readable. So while I said in the beginning you could optimize for agent not being conflicting. There is also another kind of thing you have to think about which is the human review. How much change can a human review but you can build like separate Pars in multiple main merge trains for people to do fast reviewing of those things.

Now a lot of people are focused on kind of you know when the PR and we want to reduce the number of PR reviews is that there is some kind of automated review. Now the first part is typically to reduce the failures which is great right like we we want to provide better context better tools we have adversarial agents look at the code before it comes to us that is a good way of reducing kind of the risk uh by it being better quality now at the same time we're optimizing that we can actually see what it did we are asking for screen flow uh shots for videos for proof for testing all the things that could help a reviewer and then ultimately how far do you go several people tell me like hey I'm not looking at the code anymore and I asked them okay you know but what is the impact of this going to production some documentation that is limited impact maybe you know some don't have real customers which is also great right like the impact is not a lot but if you really start thinking about what you're doing is you're selecting by what risk you can take. So some of the tests will kind of say well if it's maybe in the documentation part you can go ahead if it's like a a more impact fail change like within authentication you kind of do more kind of review by human and sometimes there's a middle ground hey this feature can just roll out we'll check what the output is whether there's any errors like gradual releases. So instead of only optimizing for quality and reducing of the likelihood, you also kind of have a way of looking at it from a risk perspective. And when it moves out of production, you know, you can also see how you can mitigate the risk, right? Because if the impact is that you uh lose data, do you have a fast restore strategy? uh or you can think about this like what if one node fails or can I fast roll back those are architectural decisions as well. So what you want to do is when things go wrong you want to have it being cheap to restore or to fix and that is important in not only optimizing but there's multiple ways of kind of dealing with the risk on the PR review.

Now in a nutshell you can do a lot of automation and you will use a lot of skills uh and that like context and better improvement uh on those pieces and you kind of reserve the people for the exceptions and that's the way that I want you to think about is automate what you can but make sure you are able to route to a human to make the right decision

with all that infrastructure with all those kind of ways of working together. Um there is kind of a new team that is forming which is almost like the factory team uh similar to a platform team that was dealing with a kubernetes or with a cluster or with the cloud but right now this is they are dealing with the coding agents and the coding fabric of working all together. They make sure that who owns the skills the context is up to date. they make sure that things are connected to the right pieces uh and that the whole factory also itself is being maintained uh and being monitored as well.

So this is a nutshell. There's a lot more to it, but this was in a nutshell two things like the pressure is building up on agents working more together in the codebase and then the humans also kind of working with the agents more and more together in a concurrent way and delivering more output at the same time. And for that there needs to be a lot of change. Now one of the things that we're playing now with is they're not just the coding agents. They're not just the people that do things in the board. Now they're being those personal agents and they will work in the uh board. They will make decisions for you. They will provide answers to other agents as well. This is a new way of building. This is a new way of kind of delegating some of the tasks that you are doing in the factory.

So my advice is brace yourself. We're not there yet. like the coding agent were a good step on your solo factories. Multiplayer factories will bring new dynamics both to the agents coding and both to the organizational working together. And that's it and thank you very much for listening. You can kind of find the slides by asking uh to me in DM on LinkedIn. I happily send you in exchange for some feedback. Um and you know you can of course try the Tesla factory where we implement and uh are working on kind of loops factories and skills uh for you uh being in a multiplayer mode and I'd also uh shared the research at a site on concurrent.cd where I find like people and uh tools and techniques that people are solving uh this concurrency all together. Thank you very very much for listening and I hope you learned some of the new things today and feel free to leave uh some comments and I'll happily have a look and respond to that. Thank you very much.