Today we're talking about RL environments and how to scale them.

But the title is a little bit of a red herring. We'll talk a bit about the engineering pieces and like running these with thousands of parallel rollouts and sandboxes on hundreds of GPUs, but I'm mostly going to focus on a different notion of scale.

Uh, and what I mean by scaling here is we there's a number of different ways we talk about scaling in the context of AI and research. We know about scaling laws and we talk about how much data you need, compute and parameters and that if you pour in more data and compute and parameters or inference time.

All of these things make models smarter or more performant.

But there's also fuzzier side of scaling which is sometimes referred to as unhobbling or algorithmic tricks or talent.

But where does this come from? It's not just pouring in resources, but it's something that is more intangible, harder to put a finger on, but really it comes from a community of people, a company, an organization, universities, the world, the internet, talking about ideas and sharing them and working on different applications, having these applications inspire ideas, using these ideas as test beds for different techniques, and building on top of these to increase the accessibility for other people in the future to not have to reinvent the wheel and to be able to build from uh what has been done by those before them to uh do more effective research and accelerate the pace of innovation.

having these applications inspire ideas, using these ideas as test beds for different techniques, and building on top of these to increase the accessibility for other people in the future to not have to reinvent the wheel and to be able to build from uh what has been done by those before them to uh do more effective research and accelerate the pace of innovation.

And so why do we have this talent bottleneck? There's a big issue that we hear all about with AI labs trying to like find more talent and salaries are going through the roof and everyone wants to hire the best and brightest AI researchers.

But one other approach besides trying to just pay the most is increase the pool.

Uh and so how do we increase the pool of AI researchers? How do we make doing AI research more accessible?

And I want to talk a bit about who we are at Prime Intellect. If you haven't heard of us, we are a bunch of things. We're a research lab. We are a comput provider. We're a platform company. And we are an open source ecosystem.

We do a lot of things and they all fit together in a way that I'm going to try to explain in this talk.

But we see these as all different pieces of how we can build a business around doing exactly this, which is increasing the accessibility of AI research and making doing research more of a toolkit available to people at organizations around the world without needing to be inside of a large lab or without needing to spend crazy amounts on massive clusters or go do a PhD.

We think that there's versions of doing AI research that really should be part of the breadandbut workflows of AI engineers around the world as we build applications and try to improve our systems and models and products.

And I think a thing people are kind of iffy about in terms of AI is whether open source models are going to work. And in my mind, that's not quite the right analogy to draw.

And so when we're comparing like AI to traditional software, there's lots of like great examples of open source software ecosystems that have been thriving in the past, things like Linux and Node and Apache.

But in my mind, the analogy in AI is not models as kind of these fixed checkpoints, but it's about research as a practice and research as a set of ideas.

And it's one that's more intangible, but there's a lot of parallels in terms of the goals of the best practices of growing a research ecosystem as well as a software ecosystem where you want to uh compound abstractions and best practices and have better tooling and iteration efficiency and have these gains over time allow uh more advanced powerful complex things to be built by uh decreasing barriers to entry for any given application and allowing this to become more accessible.

and have these gains over time allow uh more advanced powerful complex things to be built by uh decreasing barriers to entry for any given application and allowing this to become more accessible.

And so one thing that we a term we'll use to describe some of what we're building at Prime Elect is we like this phrase called the open super intelligence stack.

One because it's a fun acronym but also I think the idea of the stack of of all the pieces of the puzzle to build the engine to go do research.

Uh there's a lot of layers to it. You need compute uh you need orchestration you need libraries for doing uh training and evaluation and you need platforms to support things like code execution and eval inference and fine-tuning and we're doing all these things.

Uh but really the goal of this is to give people the tools to be able to go train models. We want people more people in the world.

And we think I'll explain why in a bit. There's a lot of reasons why uh the best products are going to be the ones that are not just kind of taking the thing out of a box of an API and putting a thin wrapper around it.

There's ways you can kind of improve around APIs. But I think in many cases people are realizing that winning products are going to be the kinds of things that whether it's a part of the model, a part of the stack, the part of the product or the whole thing, the ability to do research and have at least the option of deciding where in your product you might want to customize a model or improve a model gives you a lot more flexibility to really u make the best user experience.

the ability to do research and have at least the option of deciding where in your product you might want to customize a model or improve a model gives you a lot more flexibility to really u make the best user experience.

Um, and so we have heard the phrase in the past that the model is the product. And I think we're starting to see now this change a little bit to a lot of winning applications have the product kind of be the model.

And I think the two notable examples of this that I'm big fans of and heavy users of are Cursor's new composer model as well as uh OpenAI's codeex.

And I think these are both good examples of models that really are where the product kind of is the model very directly where the the model was trained to be the model for that product and the experience of using the model is the experience of using the product.

And the way that [clears throat] this is done is by taking a harness that represents the product and training the model in the harness in essentially an environment, an RL environment.

And environments really are just a harness with a collection of tasks and rewards.

But they also have many other parallels throughout the ecosystem. Environments are not just for RL.

Environments are also essentially the same thing as evals. Environments can also be engines for synthetic data which then you can use for SFT or distillation.

You can do RL in them directly.

But also the agents were actually deploying and monitoring out in the world. These are environments. The product of these things, the tasks, the harness, and the rewards, whether this is a data set offline or the stream of user tasks coming in to a product is an environment.

And so this as an abstraction I think is a very useful way of framing what it might look like to start having uh research become more of a a practice that is adopted more broadly beyond just large AI labs.

And I also think that there's a sense in which they're a really accessible entry point.

Uh and so I like the analogy of environments as kind of like the web apps of AI research. And what I mean by this is that they're very simple. They're self-contained.

They can they start simple but they can also get quite complex. They can get very elaborate representing the full complexity of a large product.

They're also pedagogical in nature and that you can start simple and as you build complexity, you start bumping into these walls where you have to start learning new concepts, understanding more about scaling the system side, understanding more about the hyperparameters and the algorithms and they kind of open this door where you can by playing around with them start entering into a world of research without needing to kind of build a whole training infrastructure system from scratch.

and they kind of open this door where you can by playing around with them start entering into a world of research without needing to kind of build a whole training infrastructure system from scratch.

Um, and they also require experimentation. And so I think the key different uh differentiation between just an agent harness and an agent environment is that the environment forces you to also have your tasks and your rewards predefined to be able to do this experimentation.

It's a proper eval. And what this means is that you can't just vibe check it.

You can't just like build it and test it out a bit and say, "Hey, it's good. We're going to ship it." It forces you to say, "Okay, let's think about this a little more scientifically. Let's do some experiments. lets try out different models, try different hyperparameters.

Uh, and it also gets you to the point where you can start doing more advanced research in terms of RL training or distillation or fine-tuning.

And uh, so to really facilitate this, we wanted to make the environment as an entry point much more accessible. A few months back, we launched what we called the environments hub, which is a open source community platform for creating, discovering, and sharing RL environments and evals.

And so far, we've had a lot of fun kind of seeing everyone build here. We've had hundreds of builders and environments come create either their own ideas or re-implement papers.

Uh there's a bunch of examples here I can show you, but really it's just a bunch of people who have wanted to do research and found this as an entry point to start digging a little deeper.

Whether this was investigating some benchmark and figuring out how to reimplement it or modify it to be appropriate for an RL context in terms of like new data or new examples or whether [snorts] this is some game that they'd been thinking about or some other task.

But having this as an abstraction for encapsulating the the thing you want a model to do is a way of allowing yourself to start experimenting with ways of improving it without needing to have the answers.

So I think people talk a lot about how fine-tuning never really took off in the SFT regime. And I think a big part of this is that getting data was really hard of the actual like solutions.

I think having labeled examples of what you want the model to do is a very difficult thing to ask someone to go create.

But if you can just think about the the settings it might be in without having the answers up front, if you can measure the answers, now you kind of can start creating data on the fly.

And this engine is really what the environment is about unlocking.

Um, and so actually 9 months ago, I was right here in this room. I had just released a library called verifiers, which I'm still working on today.

Um, it's come a long way, but it's a toolkit for building these things. And it's been a lot of fun over this past year just playing with it and extending it to support more features and kinds of environments.

But the idea with verifiers is to give people a toolkit that is uh essentially a bunch of components that you can mix and match and compose to do things like from simple evals or QA or games to things like tool use or using sandboxes or agent frameworks or uh uh like CLI coding agents or math problems.

There's all sorts of things you might want models to do or agents to do. And it's a toolkit for building environments that is then uh ready to be automatically trained with reinforcement learning.

And the way we thought about this design, it's been a lot of fun and also a big challenge to think like okay, how do you make a toolkit for this stuff that actually covers all the bases?

And I think there's a lot of different approaches I've seen people go about and I I think they all make sense depending on what sorts of things you're wanting to work on.

But we took a very kind of a general approach where we tried to say we are not going to know all the answers right away. There are going to be lots of pattern. There's going to be lots of special cases. There's going to be hierarchies of complexity. There's going to be patterns.

And we really wanted to prioritize extensibility. So we think about these things hierarchically where let's say you want to do a a coding agent environment for client bench. uh this which is an instance of the harbor framework which is a example of a CLI agent which is a multi-turn environment which is an environment uh similar for text arrina and Wordle or for search with MCP or for giving a model a Python ripple in a sandbox and so thinking of these things hierarchically allows us to kind of really determine like what are the foundational pieces what is generic across all environments and then how do you build up the stack towards applications

let's say you want to do a a coding agent environment for client bench. uh this which is an instance of the harbor framework which is a example of a CLI agent which is a multi-turn environment which is an environment uh similar for text arrina and Wordle or for search with MCP or for giving a model a Python ripple in a sandbox and so thinking of these things hierarchically allows us to kind of really determine like what are the foundational pieces what is generic across all environments and then how do you build up the stack towards applications

And so for one like example of this that I'll kind of walk through the whole process end to And we we call this one wiki search, but it's basically a simple search setting where we give an agent the ability to uh call some tools to search over Wikipedia pages and find some answers.

And so here is the environments hub page. So the environments hub is a kind of full stack uh code management package registry.

So every environment is a Python project where you can have dependencies and versions and uploading your evals and whatnot.

Um but the environments are very simple. They start simple. They can get really complicated, but this one's pretty simple where we just kind of define our tools as async Python functions.

We have our data set and we have what we call a rubric.

And so a rubric is the abstraction for managing the different pieces of your rewards where you can kind of compose different things. You can also have metrics that are just a zero award but are for in uh observability of what's going on.

And then the other piece of doing training will be a config. And so the config here is for our prime RL trainer, which is our kind of large scale training stack, which has been our uh culmination of all the best practices from the research literature for large scale asynchronous RL training.

Um, but the config files are intended to expose kind of the pieces that people need to think about in ways that are starting to get you more into the algorithm, but are also still designed to be pretty high level, pretty self-contained, and with with defaults that we think are going to be sensible for a lot of people.

And so running this is just kind of running a command line with uh you specify the environment and if it's in the environment hub it'll automatically install it and start your training run and then you can if you're lucky see your reward curve just shoot right up.

Um and sometimes it doesn't go this nicely but the process of doing this is iterating on your environment on your rewards and your data and your tasks to understand what makes this task holistically actually tangible in practice.

How do you tune the parameters? How do you look at your data? How do you define your rewards?

Uh, and if you do this right, you can get really good improvements, especially from really small models, but also for much larger models.

And so in this example for the the wiki search one, we started with a a Quen 3 4B model, which was about 55%.

And after training, it was at 89% on par with uh much larger models like GPT4.1 as well as reasoning models like uh GBD5 mini.

And so I think this practice of taking small models and being able to make them much better is a big win for a lot of applications where you either you want a really fast model, you want a really cheap model, you want a really really powerful model because the best models out there just aren't quite good enough.

These are all the different things you can do with model customization. And this practice of doing of creating environments isn't only for customization, but it gives you this option.

And so if you need to do eval anyways, it's useful to think of them as environments because the environment opens a lot of doors for whether this is prompt tuning or whether it's model selection or whether it's just getting a better sense of how your system could work at scale with many many users in parallel.

It's a design process that really forces you to kind of pin down what is the thing I care about? What is my agent? What is my product? What is my harness? What am I optimizing for?

Um, and so to kind of fully stress test this, we've been training a large model which will be out into the world quite soon called Intellect 3 with our full primal L stack.

And this has been us really kind of validating the efficiency and performance at a very large scale. So this is a 100B plus model trained on 500 GPUs where we've kind of done the endtoend uh post train of SFT and RL which the primaril stack also has SFT if people want to do that.

But it's also been about just understanding all the best practices. We love reading papers and we try to kind of try out all the tricks and see which ones work and see which ones don't and then distill this into a library with Primaril that can then be kind of consumed by the end user without needing to do all this uh implementation themselves.

And so for us it being open is very important. So Primaril is on GitHub. You can go find it. Verifiers is on GitHub if you want to check it out.

And for us, this is really about opening the door for more people to start learning about these things and for incorporating it into their workflows for optimizing their models and their products.

Um, and [clears throat] the only way to do this that we've what we see as the best way to do this is through growing community.

And so for us, it's been really important to really think about getting good feedback loops from the people who are building with this and understanding what they want, understanding what's going well, understanding what's painful, and addressing those problems.

And so we've done a number of community programs in terms of sponsoring different kind of small tasks to uh a research residency program with uh grad students around the world uh and collecting like uh a smaller subset of the environment hub ones where we'll actually review them manually.

And so this repo here, the prime environments repo is the ones where we are doing these directly where we're kind of offering to look over someone's kind of example.

And so we've had hundreds of these come in and there will be hundreds more.

And uh it's been a great learning process because it's forced us to fix a lot of things. We kind of understand the rough edges. We understand what we need to add.

And we're kind of then distilling [clears throat] all of these learnings into what will be our kind of upcoming uh platform product which we're calling lab.

And the idea of lab is to give people an interface, a platform where they can browse environments, they can run their evals, they can do their inference, they can do their fine-tuning and they can have research be more accessible in a way that it hasn't been historically because I think a lot of people find infrastructure very painful.

They find dealing with torch versions painful, flash attention and VLM and getting all these things to work.

We are happy to do that, but we understand that a lot of people may not want to.

Um, and so the idea with this is that if you want to go read the code, you can go read the code, but you don't have to run it. We can run it for you.

Um, and so this has been our version, which will be kind of out into the world in the near future of trying to allow people to really focus on the environment where the entry point to lab will be the environment.

If you want to do synthetic data and SFT build, let's build an environment. If you want to do your evals, you build that as an environment. If you want to do RL, you build an environment.

And I think building an environment is the kind of thing that I imagine a lot more people are going to want to be doing as we start really seeing where models are headed.

In some cases, this will be we're going to use fine-tuning services from the labs because they're going to offer this because people want it.

In some cases, this will be we really care about the smallest model we can run on prem at the lowest latency and we're really just going to optimize for our one thing.

or it could just be research for the sake of research and advancing our kind of collective understanding of how this stuff all works.

And I think that's really our goal is to have a world where there's going to be a lot of AI and where we can all kind of talk about it and understand it and look at it and poke at it and tweak it and have a better sense of what we're actually building because I think there's a lot of times when it feels like we're just kind of the model is a black box and digging into the research and going under the hood and changing things and breaking things tells you a lot about how these models work.

because I think there's a lot of times when it feels like we're just kind of the model is a black box and digging into the research and going under the hood and changing things and breaking things tells you a lot about how these models work.

It tells you a lot about understanding where they came from, where they could be going, where they might be headed, and preparing for that future. Thanks.