长周期多智能体系统的自动化研究：用闭环实验突破‘提示词微调’瓶颈

[Speaker 1]: hi, ron, i'm arena former engineer at microsoft and supercell. and today, i want to talk about autor research in a multi agent AI village. i ll use a video game like AI village as a running example here. but the broader question is one, i think many AI engineers are starting to run into. how do you evaluate an improve agents that carry state over long period of time? Speaker 0: 嗯, [Speaker 1]: before i get into the auter research layer, i want to talk a bit about project pardodox develop project paraddox at supercells ai innovation, love move. and my teammate natural money condedn, we built a modellar ai framework that allows any development to plug inteintelgent autonomous agents within a vdeo game that can intertract compete our corporate with other players, our agents as well and place them. and and them them dynnamic game companions. now to give examples of what these agents can do, the agents can move with intent. they can go to any location or person and theyguided by their own memories. emotion are curiosity. these agents can in tract with the world, they can pick up objects, drop them anywhere. and they're also aware about the context in their own environment, such as objects or other characters or agents as well. i would also like to note that game of a perst can also add new actions for these agents to accomplish within our framework as well. instead of just dropping ing or placing objects, agents can also extually react to what's happening around them. and these events that happen around them affect their own beliefs and emotions on the fly as well. and of course, it wouldn't be complete if agents start art conversation, right? agents can in this a scenario approach. other agents are even the player as well, and this makes the game feel more alive. and of course, these conversations are stored within their memory and is according to their own and affect their own emotions and believes our goals as well. and altogether, these agents make our multi agentic framework. um yeah, yeah. second. so the architecture was intentually statefall behind this. the first important part was per agent memory. each agent has its own memory name space ed by irag. so many we do not bleed between agents. second, we trapped emotion as a small vector. so after an event, our conversation the system could update values like joy, sadness, fear anger are discussed. third, agents had belief scores towards other agents and the player you can think of this as a trust matrix, basically like after the interaction happens, the elym basically decides whether the trust score should go up down or whether to and change at all. and fort, every memory receives an important score um to explain this very like, let's say, you had dinner few days ago, you probably wou't remember what you had for dindinner. but but if if one was was urdered a few days ago, you definitely remember that so the agent will evaluate or the lm will evalid uh an important score of an event. and if it crosses as thasuld, it will shore that specific memory uh in a separate cash, so that important context can be retrieved better later on. and here's an example of it just working. we 're going to ask one of the characters you're going to picnic with us here. um our character blossom um decides to pick up a pastry and go to the picnic yia because we asked her to do so. keep in mind during the conversation in the background, she plans all of these sequence of actions to accomplish. and when when we talk to her afterwards, she will also reply within context as well.

[Speaker 1]: but this is where an interesting problem actually started. as you saw in the last example, like for shark term, again, play this, our architecture work pretty well like a character could make a plan move around talk and remember the recent intraction and respond to us or other characters as well. but over longer, horizons, this is very notice the social consistency start to get weaker. so in this example, we have one agent spreading a rumor about a faile on bengals to another regent, and that agent receives that information and goes and tells another agent about it later on. after a number of events of her occurred in between when the player asked one of the agents about the mangoes, it doesn't exactly store that context that we were expecting. i doesn't give us the context that we kind of wanted to. and this is where things are trying to get messy, naturally like the system may rememred the rough topic, but lose the source of the topic. a rumor may become certain. instead of just a rumor like the agent might stated as a fact or um an dient might know a fact, but failed to fail to remembered while creating a plan for its actions. so the question here became harvely imve a mulltiagentic system over long running social behavior and not just over one response.

[Speaker 1]: and this is where we wanted to bring an artter research. as you all know, uh, few months ago, karpari posted out utter research. and this this made us immediately racuous. um perhaps we can make the system run experiments on itself. and can we use this for our system as well. so what we understood it is instead of manually tuning a prompt or watching one nice demo, we could define a scenario suit, run the agents, collect traces, scored the behavior and chanined a small policy surface, and only keep the changes that actually improved the score. and this is where we're trying to bridge project paraddogs with auto research. so at this point, basically our multigentic fammer project paraddogs is more like a labinent and auterreseararbecomes the experimental loop around it. and importantly, this is not only about improving grag tritrival. the broorder framing is optimizing the agent protocol. like how do agents write memories, retrieve them, communicate uncertainty, a bit trust, active resources and repplant around new facts. basically, um yeah, in this context are yeah. 好, yeah. in this context, art research is a not another agent in the village. like i said, it's a meda system outside the village. the villagers have local perspectives. of course, they only know what they saw heard remembered are infered, because there isn't the common memory database in between them, information only travelles. once ce other agents communice than properly the art research layer has has different job b here. it reads the full traces of a run, compares what happens against the scenario ground trooth um scores the behavior and proposes a constrained tees to the agent protocol are coordintive policy, then agreed runs to scenario and ask society level behavior like, did society level behavior get better。 this is the key shift we are trying to look for. so we were no longer evaluating one answer. we were evaluating an entire run. and this is what one of the loops would look like. like first, we define a controlled scenario, which all elaborate a bit more about later. for example, one agent learns a public fact or one agent, hears a rumor um that could be a controlled scenario. then we run the simulation during the run. we collect structure, traces observations, conversations, memorites ts retrievbeliebelief objects, whatever is relevant to us. in that case, we collect, then we score this behavior did the information spread as we expected to did the source attribution survive, such as does the agent. remember who started the rumor? did uncertainly stay uncertain? did agents act on what they actually knew? and then the auteror sesearch yer here proposes a small policy change. and this is important, it should not rewrite the whole application. of course, it should only edit a controlled policy surface. and then rerun, if the score improves and the guardials hold, we keep the improvement. and if not, we simply just revert back and talking about controlled scenaris.

[Speaker 1]: the reason why uh scenario design matchers is that social behavior is otherwise a bit fuzzy y in general. in the sense, if you just let the agents in our environment wonder around, it might look cool and you might get nice interactions, but it's actually very hearkey valid on whether the system actually improved. so this is why we believe you need controlled scenaris. for example, one scenario could test a public fact division, let's say, agent a learns. um the bakery will close tomorrow, do the right agents learn it. do they remember who said, what do they do? they change their plans based on this fact. another scenocould test rumor uncertainy agent, let's say agent here that agency might leave the village. when this rumors threads does might leave suddenly become is leaving? or does it stay as might leave like does it become a fact? or does it still say, as AA rumor, another scenario could test replanning, the group has a plan, but one agent learns, let's say the roroup they wanted to take is blocked. do edents update this and communicate this with each other to avoid um improper plan or steal actions. the point is not that these exact scenaris are universal here. the point we are trying to make is that long horizon agent behavior need scenario suits and talking about our mango example, again, after running one of our archy research loops this time after uh, a long period of time when the player finds. he asked one of the agents about the salilon mangoes. we did find that the agent was able to respond within context this time like compared to last time, yeah. and for this talk, the the exact formula, we believe is less important than the shape of the score card. uh you do not want a single vae e tritric like asent quality. this will hide all the interesting failures. instead, you want a balanced score card for difusion. you might measure reach like how many agents know the fact after end steps for a provenenence, you measure source attention among agents who know it, how many remember it where it came from exeta for rumors, you can measure uncertainy preservation and false certainy rate for planning. you can measure action consistency and time to replan and for privacy. you can measure containment this matters because optimizing only one metric can create bad behavior, because let's say, if you only optimize for a diffusion, the agents may learn to overshare everything. and let's say, if you only optimize for memory ealds, you might create noise, you are still like memories. so this score card is what keeps the system honest and prevents the auto research agent from gamifying the system to just increase one specific score.

[Speaker 1]: the other import enering lesson that we learned over this project is that um it's imppart to keep the editable surface really small. the arto sesearch's alyer should not have permission to romly rewrite the whole code base. instead, it's really imppoint to freeze the harness, the scenos and metrrix. so we're only exposing the part of the system that we actually want to optimize here in project products for us. that meant tings like memory, writing policy, retrieval policy communication prompt, believe trust rules, source attribution, replanning, triggers eetra. this gives the search proprocess room to improve behavior, but it also prevents it from gaming the evaluation directly as we mentioned before. and this is the difference between the elam writing lurandum patches versus the MM actually searching within a controlled policy space. and here here, examples of the kind of changes, i want this kind of loop to search over, if if source attribution disappears, the policy change might be preserve source in memory and uright memory rights and summaries. if rumors harder into facts, the policy change might be stored confidence. mark first hand versus second hand and require heading when retailing uncertain claims. if if facts, if public facts stay local, the policy change might be classify useful, public facts differently and make edens practively share important source evidence. the key is that these are small changes to the agent protocol, but they can have larger effects on a society level behavior from monthly agentic systems. this is also where i kind of want to be careful about our claims here because we believe, without repeated to current loop results, like i won't say the system this generally improved. we're trying to say this is the right kind of surface to expose to an autorresearch layer. uh p. because it is small enough contcontrol, but still written enough to change the social behavior, to some extent, at least. and the biggest lesson for me, perhaps, was that memory is not enough here. you can add a add memory to an agent and still not get the current long term, her horizon behavior that you are looking for um because agents need to sometimes know where that information came for came from. you need to preserve whether it was first time, second hand, raified or uncern. sometimes you need to separate draw pacodic memories from what the agent and currently believes to, and you need to test behavior through scenario, not not through vives. so the other lesson is that um rollback also is not optional when you opto my social behavior, a change can improve one thing and damage another. so policy that spreads public facts um faster might also leak private information, a policy that increases recall might increase scale memory usage. so the loop should basically be like a ratred try a team score. it keep it only if the score card improves and gardielseful. and we we definitely believe this is not only relevant ant a game agents. because, although i gave an example of using a game village, we believe, like, let's say, for example, support agents support agenciy to know which policy of date comes from wewereed. and whether it's supreceds and older answer personlistence, for example, need to remember commitments that the prevacy made and make corrections, if if the user wants to change those personal commitments, research agents, proprevenence cctations, contraction handling and hypertests updates coding agents need long running context across issues, fres teammates and changing requirements. birfflu eens need access controls, handdofs and replanning. when the world changes, all of these systems have the same underlying problem, they maintain state over time, and that state affects affeacaction. so they control sceniors and behavioror score cards is what we are proposing. so again, in brief a recipe for long horizon agents. if there is one practical recipe, i want you to take away freeze. the harness defines senials log traces a score behavior and expose only a small policy surface search over these changes. keep only changes that survive your measurement, and this is an engineing pattern that we believe would make sense for long running agents. the real question we believe, is across control runs. does the system behave better to close project padct start as an attempt to make game, even n't feel alive in a trady world, but the deeper engining problem was not animation or dialogue for us. it was the state, such as which agent knows what which agent told whom what is true, uncertain are outdated and do agents act on what they remember. alter research, alter research gave us a way to approach this a bit more systematically, not by trusting one demo or not by endlessly handtuning proms, but by running control experiments and keeping only the changes that survived our measurement, long horizon, agencts need experiments and not just prompt. and i hope that's to take away that you get from this talk. and yes, please do connect with us. we'd love to talk if you have any questions. thank you so much for listening. yes.