开放超级智能：重构AI研究生态
在人工智能快速演进的当下，事后训练（Post-training: 在基础模型预训练完成后，通过微调或强化学习提升模型在特定任务上表现的过程）已经成为决定模型实际应用效果的核心环节。作为致力于降低大规模开源 AI 研究门槛的创新力量，Prime Intellect 旨在构建一个完整的“开源超级智能技术栈”（Open Superintelligence Stack）。我们希望让企业不仅能够使用优秀的开源模型，还能够根据生产环境中的实际业务流、产品和内部任务对模型进行定制化微调，实现低成本且高效率的自主迭代，从而摆脱对闭源前沿模型 API 的绝对依赖。为了实现这一目标，我们推出了全新升级的开源工具链，包括 verifiers v1 库和 prime-rl 异步强化学习框架，用以解决复杂智能体场景下的训练与工程挑战。
为了演示这一整套工作流，我们还同步发布了开发者 Cookbook 仓库，以直观的代码实例引导开发者快速上手这套现代后训练技术体系。
Original English Source
Hey guys, how's it going? Thanks for showing up. This was a little bit of a last-minute assembly. I know a few days ago I was like talking to Swix. I was like, "Hey, can I still do a workshop?" And he was like, "We have one slot left. It's Monday at 4:30." And I was like, "I'll take it." Um and uh then yeah, um I wanted to kind of just do a bit of an update on uh some of the stuff we've been building at Primed and Loaded. So, if you don't uh know me, hi. I'm Will Brown. I lead applied research at Primed and Loaded. Uh we do a lot of stuff around uh every part of the kind of AI research infrastructure stack. Uh today is going to be about post-training, which is where I spend a lot of my time thinking and building. Um and especially want to be talking about uh the post-training tools that we build uh that are fully open source, uh the verifiers and Primed RL libraries, uh which kind of go hand in hand um both on the environment side and the training infra side. Um and show off some things we've been cooking over the past few months that I think is uh kind of the way that things have evolved as the agent use cases have gotten more complex, but also kind of clearer in terms of what people want out of agents and the sorts of things that are needed to like do the sort of post-training that is needed to uh power like the real-world applications people are building nowadays. And so, broadly at Primed and Loaded, we are our goal is to make doing large-scale open-source AI research easier and to enable companies to train their own models and deploy them and have them improve based on the scenarios that they actually see in production in terms of use cases for applications and products and internal tasks and workflows. Um and to give people an option to not just use the open-source models that are getting quite good, but to take them and make them even better on their own use cases.评估即环境：启动模型优化飞轮
在建立起基础的研究共识后，环境（Environment: 封装数据、交互逻辑与奖励机制的评估与训练场景）在后训练的整个闭环中扮演着至关重要的角色。它本质上是指定模型行为的编程语言。许多团队在迈向后训练时往往存在顾虑，担心引入复杂的工具链会增加开发成本，但实际上，后训练的第一步就是建立完善的评估（Evaluation: 对模型表现进行可量化度量的过程）。评估与环境在底层逻辑上是高度统一的。即使你当前直接使用前沿模型的 API，通过环境来进行评估也是保障产品质量的必备功课。它可以帮助开发者在智能水平与计算成本之间寻找最优解。一旦建立起评估体系，这就自动解锁了模型的“进化飞轮”：利用收集到的真实世界反馈不断修正评估环境，进而驱动模型持续迭代，让训练算力转化为长期的推理边际收益分摊。
在后训练的具体策略上，业界传统路线是先进行监督微调（SFT）再进行强化学习（RL）。不过，同策略蒸馏（On-policy Distillation: 将大模型或多个专家的能力蒸馏到较小模型中的过程）和自我蒸馏（Self-distillation: 模型通过自身生成的高质量数据进行自我迭代优化的过程）已逐渐成为主流。特别是在训练多任务专家模型时，一种非常有效的做法是在同一个基座模型上针对不同环境分别训练多个 RL 专家模型（RL Experts），然后再将这些专家的能力共同蒸馏到同一个最终模型中，这能极大提升训练的稳定性和最终模型的泛化能力。
Original English Source
And so, we use the phrase the open superintelligence stack to describe what we do... and showing off some of these new things, but all under the umbrella of what does modern post training look like? What does it mean to kind of take a model and train it to be better at your task? What are all the parts? What are all the kind of gotchas? And how do you orchestrate this into a system that is actually easy for people to use without needing to go build a massive research team and to be able to kind of have it be accessible in the sorts of things that anyone who's an AI engineer at any like startup or enterprise that wants to invest in post-training can actually do. And so there's a cookbook repo that is that's kind of like an alpha release right now. It's still changing a bit, but it's a preview of kind of all the stuff we've been building over the past several months. And so today we'll be kind of following along that framing a good bit. And so I think the first thing we'll talk about is just kind of what is an environment. People talk about environment in the context of RL and they think of like RL environments, but environments are more than just for RL. They're for all sorts of things in post-training and evaluation. We're going to talk about what we're going to call the the V1 version of the Verifiers library, which is a full overhaul... As well as kind of talk about how Primer RL has evolved as a library. And so Primer RL is our like full-stack open-source training framework to support asynchronous reinforcement learning. And we've got a lot of fun new bells and whistles to show off in terms of both scale and features. A lot of this is in service of custom algorithms. So making it much easier to do the kinds of things that people are interested in for modern post-training. If you have been following the news on on policy distillation or self-distillation or all these other fun new algorithms that people are coming out with. It is the the age of research indeed. And we don't want to just like train small models. We want to train big models. We want to train them really efficiently because as models get bigger, the compute starts adding up. And if you want to make this accessible to people, especially if you want to be able to iterate on it, it has to be fast. It has to be cheap. It has to be affordable and reliable. And all of these kind of funnel into our lab platform. We'll talk about both some of the things that we've already released there as well as some things that are coming soon. And so the post-training loop in my mind kind of revolves around environments in the sense of environments are a language for specifying what you want your model to do. Um, they are an encapsulation of the data you might have, the scenario you might want your agent to be in, uh, the way it'll interact with that environment, uh, as well as how to to score what good looks like, to determine what was good and bad. Um, and often this is the first thing you want to do with an environment is just evals. And so I think a lot of people are maybe nervous about getting into post-training. They're like, "Oh, it seems like a lot of work. There's a whole new tool chain. Um, what if I'm already using like the frontier models and I want good results out of them, uh, or I'm getting good results out of them, or I want to like see what I can do at the harness level first or prompt optimization." And that's all good. Like we're not necessarily asking people to just like throw everything away. I think in many cases what people will find and what we see with our customers is that um, the systems that work best for them involve using both. And you kind of want to be able to make these decisions about what is the right where is the right place to train, uh, where's the right place to use a frontier model that is available via some API. Um, and so evaluations are kind of very key to this. And so like evals are the thing that opens the door to post-training. And so environments and evals are essentially the same thing. Um, but once you have evals, now this is the same kind of unit of, uh, like logic that you actually need to do post-training anyways. And so, uh, building evals is like just good for your product hygiene no matter what you're doing. If you want to kind of decide whether to use GPT or Claude, or decide do you need Opus or uh, Sonnet or Mythos for a task. Like if you want to min-max on like intelligence versus dollars, um, evals are a very good way to do this. But evals also then unlock this this flywheel. And in terms of modern post-training, I think historically people have done SFT then RL as like the main uh, frontier model recipe, although on policy distillation has certainly found its way into a lot of workflows. Um, and I think some people are also very eager about algorithms like self-distillation. We can talk a bit about that and when it makes sense and when it doesn't, but uh, in particular one area where it does make sense to do like the whole on-policy distillation thing is when you're training experts where you have multiple different things you want your model to be good at, and people have found that if you have a bunch of different environments that are all different things, and you want to have one model be really good at them, a nice way to do this is train individual RL experts on top of the same base model and then do distillation from those teachers into the same uh, checkpoint.解耦重构：Verifiers v1的三维架构
为了支持越来越复杂的智能体（Agent）应用，Prime Intellect 对 verifiers 库进行了彻底重构，推出了 verifiers v1 版本。在旧版设计中，多轮交互和工具调用的逻辑全部由环境内部的单一循环控制，这在智能体需要使用终端命令行或执行复杂推理时显得捉襟见肘。因此，新版核心架构将环境解耦为三个相互独立的模块：
- 任务集（Tasksets: 定义具体任务的后端状态、输入数据和评分逻辑，且与具体的交互界面无关）。
- 治理器/指令器（Harnesses: 决定模型如何解决任务的程序，例如经典的 ReAct 循环，或者像 Claude Code 这样的命令行智能体，甚至是基于 LangChain 或 DSPy 构建的自定义控制流）。
- 运行时（Runtimes: 智能体执行代码的物理或虚拟环境，如本地 Docker、远程沙箱层 Prime Sandboxes 等）。
在这种全新的装饰器设计和基于 Pydantic 的强类型配置体系下，开发者可以自由组合不同的任务、治理器和运行时。我们甚至彻底废弃了原有的 Rubric 模式，全面改用可编程的配置和自动类型校验，在编译期和初始化阶段捕获潜在的类型错误。这使得我们在集成如 Wordle 游戏、基于文档检索的评测，以及需要在终端中执行代码的 Harbor 等复杂 Benchmark 时更加得心应手。
Original English Source
Um and then once you have this, you want to deploy the trained model, which could be a full uh, base model with a full weight training, or it could be a Lora adapter, and you want to serve this at scale... but really the goal is going to be thinking about what do these tools look like? How do you actually do this? What are the parts? Um, and how do we build it? Um, and so environments as evals, uh, Uh, what is an environment? Um, I think it's useful to decompose environments into tasks and a harness. Um, and this is foreshadowing some of the the refactoring we've done in verifiers over the past months. Uh, if any of you have used the verifiers library before, you may be familiar with the the multi-turn environment pattern or tool environment pattern where there's kind of one loop that is owned by the environment that you can plug in various tools into and that was really great for a very long time for getting started for people, especially back in the day when people were mostly just trying to graduate from single turn into multi-turn tool tool use. Uh, but what we found and as we kind of iterated on different patterns and extended uh with NeMo Gym and Open Ended. And most other tools that you see out in the wild that are kind of under the umbrella of uh an RL environment, we would call these a task set. Um we generally have found that it's useful to have these be harness agnostic where they represent the the back end of the server or some state that you're interacting with, but they don't own everything about what the model is doing. And so, it doesn't In some cases, it doesn't make sense to plug a model into a task set. Especially because we're kind of gravitating towards an agent world where everything is running in a terminal or it has skills or it uh is using CLI tools. Um and these things like often look more complex than just basic loops. Um but we also want to support basic loops. So, we want to kind of allow both the old way of doing things and the new way of doing things. And so, uh everything that was the old way is now the default harness where it's system prompt and tools in a loop. Um but the harness pattern also supports much more flexible execution of things like recursive language models or CLI agents like Codex, Cloud Code, Open Code, or classics from the research literature like Mini Sweep Agent or building your own with arbitrary Python libraries like LangChain or DSPy. Um and so, we've been able to decouple these into a pattern where you get to write your harness independently of your task set. Uh there are kind of some basic sanity checks about properties that like harnesses either do or don't support and task sets do or don't require. Uh and these kind of click together. And the runtime is where this executes. And so, we've uh still been embracing a lot of the async IO patterns from before, but we've leaned a little more into having things be subprocesses uh where you can still run everything locally. You don't have to use sandboxes, but you can use local Docker, or you can use our own Prime sandboxes layer. You can use any other sandbox layer you'd like or kind of build from scratch. And so the harness the runtime back end just is a place where the harness can run its code. And so the harness just needs to be able to run code somewhere as a script essentially. We've used a lot of the UV tooling where UV script is a very powerful pattern to be able to kind of mix and match and kind of contain dependencies. But what happens is once you plug these together, you run a rollout on a task from a task set and you get a trace. This is live on the verifiers main branch for prim and elect AI / verifiers on GitHub as well as it's released as a dev release. The stable main release will be kind of coming to PyPI any minute now, but you can install the dev and play around with it if you want. And so what do these look like? So tasks are just like a row of a data set. And the very basic version of it is you just start loading a data set from hugging face or anywhere else. And so the new pattern here is from verifiers V1 I just to keep the old stuff separate. The old stuff still works just fine, but this is how we have been able to kind of decouple and iterate on the new version. As well as we've really embraced this decorator pattern. We found it to be very useful. We also if you were a rubric fan, we killed rubric. Didn't make sense anymore if you were using old verifiers rubric patterns. But still it's you have functions and loaders. We are very heavy on PyDantic so everything is super typed. We have lots of powerful config features where you can have everything in a toml file. You can override it in the CLI and everything is kind of clean and guaranteed to kind of type check at like validation time rather than waiting for something to fail later down the road. And so examples of this are things like sweet wrapper where you can do a genetic code search. You can do the classic games like Wordle. You can do search over documents with judges. You can do complex things like harbor that support a lot of popular benchmarks now that need agents running in a terminal.差异化奖励：解决模型“长篇大论”的顽疾
在强化学习中，奖励函数（Reward Functions: 用于向模型反馈其输出好坏的评分函数）是引导模型行为的指挥棒。然而，在实际训练中，如果缺乏合理的约束，模型往往会演化出一种“过度思考”的作弊倾向——生成极其冗长但效率低下的思维链（Chain of Thought）。为了在追求正确性的同时兼顾效率，我们需要引入群组奖励（Group Rewards: 一种在强化学习中通过对比一个批次内多个不同样本的表现来动态确定奖励幅度的机制）。
传统的 RL 框架通常假设每个采样序列是完全独立评估的，很难实现需要对比的复杂逻辑。但在 verifiers v1 中，群组奖励是作为一等公民进行设计的。例如，为了实施简洁度加成（Conciseness Bonus）或长度惩罚（Length Penalty），我们无法预先设定一个绝对的长度阈值 $N$，因为不同问题的最优解答长度各不相同，且随着模型变聪明，其长度分布也在动态变化。因此，我们利用 RL 在同一个提示词下采集多个样本的特性，通过计算这组样本长度的方差（Variance）来进行相对奖励分配：在所有回答正确的样本中，给字数最少、效率最高的样本予以额外奖励。这种基于群组内差异的动态对比机制，在多目标优化的博弈中证明了其独特的有效性。
Original English Source
And all of these are going to be combinations of the the the task that pattern with uh pick your own runtime and pick your own harness. Um and so rewards and metrics I think are also kind of uh just functions that take in the kind of records uh of what's happening in a rollout and return numbers. Uh rewards are the main thing that'll drive progress in RL. Um metrics are just kind of like logging uh what has happened. So counting tool use and counting errors. Um these sorts of things are very useful to be able to expose in your dashboards. Um and then group rewards. I think this is something that we have fought hard to kind of make sure still is first class because we see it as very important to um a lot of the research pattern people want to do, but I think it's also ignored in a lot of like uh tooling out there. Where in in many RL frameworks it's actually quite hard to do group rewards because things are very decoupled and things kind of assume that all rollouts are going to live independently and that they don't need to talk to each other. But there's a lot of things where you really want to do pairwise judging or you want to do ranking or you want to give a bonus to the uh the shortest correct answer uh in terms of tokens used. Um and so these sorts of things are really flexible uh in terms of the we really design for flexibility in supporting the the things that we see as like the most exciting papers we've read or all the algorithms that we think people may want to innovate on um while still allowing people to have like the core primitives that they kind of expect out of an RL framework. Um and so like in group rewards, I think this concise this pattern is one that I find very useful a lot. I think um like a big pattern that comes up a lot are doing post-training is uh models will love to like think and think and think if you let them. Um and if you don't give them some kind of pressure to like be more efficient, uh I think a lot of people will notice that like open models often have really really long chains of thought. Because on one hand it's like this is a useful strategy for a model, but it's also the sort of thing that will grow like out of control if you don't counteract it. Um and so in reward design like one of the big things people will want to do is uh something like a length penalty um or a conciseness bonus. Um, and so one of the reasons this is tricky is because you don't know the optimal length for a problem. Like if I give you a math problem, I could say, "Oh, solve it in less than N tokens." But also like who knows what the right N is. It's also going to change as the model gets smarter over time. It's going to be different for every problem. And so you kind of can't know this up front. And the only way you can do it is take advantage of variance. So one of the nice things about RL is you have multiple samples typically. Um, and this allows you to use the fact that you have multiple samples to shape the reward. Um, and so if you have multiple rewards in a group, what you could do is uh look at all the ones that were the correct answer or just all the ones in general and give a bonus to the ones that are the most concise. Where if you also have a uh a correctness reward like these are going to uh ensure that um you're both incentivizing correctness as well as incentivizing efficiency. And so juggling multiple objectives simultaneously is kind of one of the hard challenges in RL in reward design. Um, but doing things like group level comparisons and kind of these sorts of bonuses are are quite useful in many cases.状态化桥梁：Renderers与Token精确控制
在智能体执行多轮交互和并行分支时，底层的 追踪图（Trace Graph）控制面临着严峻的挑战。为了在支持复杂子智能体和并行树状分支的同时，严格控制强化学习所需的线性序列依赖，Prime Intellect 独立推出了一款名为 renderers 的 Python 库。
它的核心目标是重构现有的聊天模板（Chat Templates: 规范多轮对话输入格式的模板）。在传统开发中，使用 Jinja 模板非常痛苦，微小的换行符差异或模板剥离行为都会导致训练端与推理端出现 Token 级不匹配（Token Mismatch）。由于文本到 Token 的映射具有“多对一”的歧义性，这种极其隐蔽的数值不匹配往往会造成严重的训练发散，使模型偏离原定的策略。
renderers 库将聊天模板转换为了可编程的构件，能够精细化管理“文本入、Token 出”的拼接逻辑，即使多轮对话在文本空间逻辑上相同，也能确保在 Token 空间里的一致性。虽然业界曾经希望将大模型 API 完全设计为无状态，但在处理复杂的 agent 级长链条逻辑时，管理好这种带有状态的 Trace 和 dual-stream（逻辑文本流与底层 Token 流）映射是无法规避的技术硬伤。
Original English Source
I also want to talk about like tools and user simulators, which I think have been uh becoming more important uh in a lot of complex applications where you have models that are In many cases there's like a core agent harness, but there's also In many cases you are putting a model in a setting where it's going to be in some task where a user is giving it additional tools... Um and so one of the fun things behind the scenes uh is what we call the trace graph. And so we had kind of been having this grow out of control in terms of the old way of doing things and we decided this was another opportunity to like really overhaul our system to like have really good support for sub agents and parallel branching trees while also still preserving the kind of linear sequential dependencies that you need for RL with uh careful token control. Um and so here uh there's a a notion of a of branches that are kind of like at the message level. So conceptually um the things that matter logically in environment space and in harness space are messages which are just text. Uh the harnesses don't think about tokens, uh but if you've done any RL experimentation you may have uh encountered issues where uh re-tokenization or like some messages if a model will say something and you turn it into text and you put it back through tokenizer, it can change a little bit. That because tokenization isn't is many to one. And so this causes lots of very subtle numerical problems, especially late in large scale training runs. And so you want a really nice back and forth between uh messages and tokens. And so the trace data structure that we created here partly is to enable this where we can store things both at trace level and then map them back into token level in the right sequences as needed. And we also released a library called renderers recently, which is a standalone toolkit that anyone can use that we have found the sort of thing that we're working with some of the inference tooling to support. So renderers are really all about like essentially rethinking tokenizers and chat templates where behind the scenes it's just making calls to the tokenizer, but chat templates if people have spent time debugging with them, it sucks. Ginger is awful. It's very very painful and there's so many subtle things that we kept running into where like a model would sometimes have an extra new line and the chat template would strip it out and this would like cause a mismatch in your trainer and inference that would either force you to go off policy because you now have a trainer and inference mismatch or it would cause a logical branch where a thing that is a branch in like uh it becomes a branch in token space even though it shouldn't be in logic space because of tokenizer subtleties. And so renderers as as an abstraction it was kind of pioneered by OpenAI's harmony with the GPT- OSS release and used prominently in thinking machines cookbooks as well for tinker, but we found it was useful to just kind of make it a standalone thing. And so this is just a Python library that doesn't depend on any other prime stuff. You could use it with any inference engine you want just as a standalone thing that is really designed for being able to manage this token in token out concatenation without thinking about it too much yourself because we kind of turn each of these chat templates for these the popular models into programmable artifacts where you can do things like look up a history of secret you can use the kind of history of a trace to be able to understand like what is the right tokenization? Like do I essentially have like a logical prefix hit in message space even if I don't in tokenization space after re-tokenizing? And so this is the sort of thing where I think people have gone back and forth on like whether they want LM APIs to be stateful in general. I think a lot of people were hoping that like we could just have every model API be stateless. I think maybe people are less concerned about this now because we're moving towards this agent world where agents themselves are going to be stateful APIs. But I think this has revealed to us like going through all of the the things here like why opening eye responses decided to be stateful. There are some kind of like unavoidable issues that kind of come up when you're doing large-scale agentic rollouts where you you do need to kind of manage this very carefully and it's kind of unavoidable just because of how tokenizers work. And so you want to be able to maintain these dual streams of the logical text and the the tokens and you kind of want these to be cleanly interoperable where users don't have to think about the tokens very much but the trainer gets to see everything nicely in token space as well as the inference engine. Um, and so from the harness interception server we have clients that can be used both for training and inference. And so you can kind of like swap between these modes without thinking about it because certain models like don't need to in in a training setting you need to be able to get log probs and set the temperature. Some model APIs won't let you do this. They won't return log probs because like open eye models with reasoning like won't show you the reasoning trace so there's no way they give you the log probs for everything. And so like that's fine, it's just eval only and so we have this this client layer where you can go between eval and train to be able to support all these models, but we still use the interception server pattern either way because it allows us to have like this notion of a dialect where like you can choose OpenAI chat completions or responses or Anthropic and all of these are kind of easily supportable as just like translation layers between a raw request into something that'll get passed through a renderer potentially if you're on the train client side and formatted into a a message via tokens. Um And so this brings us to Primer RL. So Primer RL is our training framework that is consumes the environment. So once you have an environment with your task set and your harness and your interception server and your runtime and your renderer and all those things, this plugs into what we call the orchestrator. And so Primer RL has been async from the ground up. Uh so I think async RL is one of those things that I think people were kind of one foot in and one foot out and a lot of training frameworks you see them uh will still kind of support synchronous training. Um some people I think have their reasons for wanting to do synchronous training. I don't agree with them. Um I think kind of want to bite the bullet of the off-policyness anyways for reasons that come up with agents um in terms of you want to be able to overlap long rollouts and not always be waiting on your slowest rollout. And this kind of means you can't be fully on policy unless you want to kind of accept always waiting on your slowest rollout. Um and so this is really why we went all in on async. And so the orchestrator's job is to allow the inference and trainer to just be separate processes, separate servers. Uh they don't share GPUs. They don't really know about each other all that much. They just consume from each other. Um the but the orchestrator job is to really like manage the run. And so the orchestrator uh will make sure that the environment is running with the endpoint mapping to the inference server. It'll do rollouts. Uh it'll package these up into a batch. It'll send this batch to the trainer and it'll be up to the trainer to figure out what to do with the batch, uh which will be kind of printing some uh sequence to feed into a loss function um based on the the specification. Um and so the the server pattern we use for environments is just an engine that can like send requests to inference and like send batches back to your trainers. It's very client-server. Um and we found that this is just a really useful way to uh allow scaling concerns to be decoupled as well. And so like for example, you can have a lot of environments running or you can have one environment running. You can have um a bunch of infra replicas, you can have one infra replica. Um you can have sandboxes or no sandboxes. Uh and the trainer doesn't care about this, the inference doesn't care about this. It's just separation of concerns at a system level uh allows you to kind of not really worry about these things as异步算力架构：PrimeRL与托管微调平台
有了环境、运行时和渲染器之后，最终拼图是将其接入训练器的协调器（Orchestrator）。我们的强化学习框架 prime-rl 从底层第一天起就完全拥抱了异步强化学习（Asynchronous RL: 采样生成与参数更新过程异步解耦的强化学习计算模式）。
与坚持同步训练的观点不同，我们认为在智能体场景中必须接受异步带来的策略偏离（Off-policyness）。因为智能体执行复杂任务（如运行测试、代码搜索）的用时差异极大，如果使用同步机制，系统会被耗时最长的那个 rollout 严重拖慢，造成算力极大的浪费。在 prime-rl 中，推理服务器和训练服务器完全是物理隔离的独立进程，仅通过高速网络进行批数据传输。协调器负责维护这一数据流泵。
此外，我们还构建了配套的云端托管平台。目前已上线了多租户 LoRA（Low-Rank Adaptation: 通过低秩矩阵分解大幅降低微调参数量和显存占用的微调技术）训练服务，允许多个用户的专属 LoRA 适配器共享同一个底座模型权重，极大地摊薄了显卡占用并支持按 token 计费。而即将发布的全参数微调（Full Fine-Tuning）托管服务，则可以让开发者在本地 CPU 上开发调试环境，再以包的形式推送到 GPU 云端。系统会自动处理算力伸缩、断点容错、沙箱隔离以及统一账单，让开发者只需专注于调整 reward 逻辑或微调参数，彻底免去了底座显卡运维的后顾之忧。
Original English Source
your own that look like adding in a function to like assign advantages uh to a rollout. Um Uh yeah. And so then then from your algorithm then from your configs you can just kind of say hey I want like this algorithm and it's just going to pick one from the registry. You can add your own to the registry if you want. Um and then this will be the algorithm used for your training run. Um and you can do this on a per environment basis if you want. Um but all of these algorithms that people are looking at kind of fall into this like table where there's questions about like what are your where are your rollouts coming from? Are they coming from your current policy model or are they coming from some other source like a teacher? Um and so any algorithm people call like on policy or like slightly off policy in terms of the async RL stuff. This is one where like your your actor in RL sense is going to be the model you're training your policy. Your policy is your actor. Um in other cases you're doing stuff where your your actor is some other model. So if you're doing context distillation or you're doing SFT, like these are ones where you are uh going to have some other like model or prompt potentially be the teacher that is generating the data that you're going to be training on. And so all of these kind of fit within this family. Then the other thing is the advantage. And so the advantage is just like if you generalize it to like a score, like you can now call like just cross entropy loss or negative log likelihood is like everything has like advantage one. Um there's a lot of all of the like OPD algorithms can kind of you can kind of think of it as the log probability is being the advantage. And with RL like your your reward minus some baseline usually group mean or something like it is your advantage. And so we just kind of like took a step back and looked at all these things and we're like, "Oh, we can just kind of like factor this all out pretty nicely." And have almost everything else be shared but these things get kind of swapped and so your infra doesn't need to change just because your loss function needs to change. And so for on policy distillation like it's just plugging in with a different loss target and it's like talking to the teacher and being like, "Okay, the thing I need to get is reference log probs from some teacher and I already have my like sequences. So I just need to send these to a teacher as prefill in terms of like a like you can get prefilled by just like asking for like a one token response. Now I I get my sequences and I can just like stick these in as my reference log probs." For self distillation you can have like a hint that you're kind of putting in before your your like this teacher. So you're not saying the same prompt you're saying a different prompt using renders to kind of pack these together and then you get back the same like sequence that you can slice out into the original form so that you can kind of add these as your reference log probs there. You could do echo where you have like two different algorithm components. You could have one that is targeting the uh the doing cross entropy on your environment tokens while doing like an RL objective on your your action tokens. And you can mix and match these. So you can have like different like you can decide which teacher is going to be done your you're going to be sampling from your student or your teacher. Um you can like decide which algorithm you're going to be using on a per environment basis. Um and you can have like this one we have like both um a normal like OPD as well as GRPO. Um and then Ah yes. So like this whole family we now support within Primerl natively. Um and one thing you may also have poked around at or seen or tried is uh we have our hosted training platform uh which means you don't have to worry about GPUs at all. Um And so this is just hosted Primerl. The version we have that is kind of the the broad uh self-serve version today is multi-tenant Laura um which is focused mostly on RL. Um what we have coming quite soon that we'll be rolling out is full fine-tuning um which supports changing as much as you want in Primerl in terms of the model and everything else where we still give you all the same abstractions for uh not needing to think about the GPUs and kind of auto-scaling and uh magic restarts and uh having a dashboard where you can log everything and have unified billing for sandboxes and judges and all these things. Um but also you get to develop your environments on CPU on your laptop, push them to the platform as environment packages, and specify them in your configs. Um and then this allows you to um have a lot of flexibility in like deciding when you want to go to different levels of the stack. So in many cases you don't actually want to change anything in the trainer. You just want to change your reward function. And you can do this in environment space. Maybe you want to configure the trainer but not change it in which case you want to kind of poke into like slightly more complex knobs for uh things like different loss functions or learning rates. Um additionally you may want to like actually go deeper into the trainer and like uh do new algorithms at the environment level at the algorithm class level or at the loss function level or maybe something else entirely that needs going even deeper. And so we we support all of this with the the um the full fine-tuning as well. And so uh multi-tenant Laura, if you're unfamiliar, is a a very useful pattern for allowing multiple people to do training runs on the same architecture. Um on the same uh model weight copy. It's so you have a base model that's kind of This is how all inference for like token-based pricing usually works as you're doing multi-tenant inference where you have one big copy of Claude uh that is just getting everyone's requests and hitting a shared uh like KV pool uh that is managed um with This is really nice with Laura because you can just have multiple Lauras available that you can hot swap and each person can have their own Laura without needing to replace the base model. Uh and you get one kind of inference pool that serves everybody at once even if they're using different models. This allows you to do things like token-based pricing uh and not need to reserve GPUs. Um For full fine-tuning, it kind of does have to be GPU-based. Um but we've been getting a lot more GPUs and so we have the ability to let people run stuff on their GPUs. Um and uh this allows going like pretty like going to full parameter training where especially for algorithms like large-scale SFT or mid-training, um you will want to kind of have more than just a Laura adapter um as well as the ability to kind of really customize at at every layer you might want um like how your algorithm is going to work. Um but so we The multi-tenant Laura one is already live and you can go use it today. Um full fine-tuning is coming out in the next coming next couple weeks. Um the V1 stuff from before is already uh out uh as a kind of a alpha feature that is a stable release coming the next I don't know, quite soon hopefully. Um and then the cookbook I mentioned earlier is also out. Um And that's mostly what I want to talk about. Um bit informal and I can People have questions, I can uh dig into any individual parts people think are curious. We're also uh we're hiring. Uh we are small team based in mostly San Francisco um becoming a much larger team quickly. Um and uh we'll have some other exciting news about that coming later this week uh to I don't know. Uh demonstrate our commitment to scaling the team. Um but yeah, we are we're growing and we're I think in a unique position of like we do very open research work like all this code is on GitHub if you want to go play with it but we also like our a real company that trains big models and makes money. And so yeah, we think we figured out a good business model to do real open research and it's been an incredible journey thus far and would love to have excited passionate talented people join the team.📌 文中提及的人物和组织
公司/组织: Prime Intellect