CTO: It is great to see all of you here today. We are super excited to talk about one of our favorite topics here at Theta. Before we get started, we just want to introduce ourselves. So, hi, I am a co-founder and CTO at Theta Software.

Rayan Garg: Hi, I am Ryan. I am a co-founder and CEO at Theta Software. Prior to this, I was previously a founding engineer at Deep Silken where we did research into ternary models.

CTO: Awesome. So, I can get us started with the topic today. We are going to be talking about RL environments within the context of long horizon tasks. And I think the most important thing for us to start with at the beginning is just talk about the trends and what long horizon actually means. So, we all know that the horizon at which AI agents can work autonomously is accelerating really fast. This is just some of the metrics that you can look at to see how this progress is really accelerating. But I think it is really important to actually define what the time horizon here actually means. We have gotten this data from one of the most common benchmarks out there that you have probably heard of for time horizons, which comes from METR. And METR has one response or answer to this really important question about how we actually define long horizon. It is really important to understand because what we considered long horizon a year ago probably is not really long horizon in our definition today. And what is long horizon today probably won't be long horizon in a year or two. And I think that gets to our first point, which is that long horizon is really a scalar metric. It is useful for measuring relative tasks, like one task might be more long than another, but it is really hard to define into a binary category of "this task is long and this task is not," especially as the scope changes over time.

CTO: The first way we can talk about defining this is how METR looks at it, which is human horizon, meaning can we use humans as a benchmark. If a task takes humans a certain amount of time, so if AI agents can do that, then they have reached this certain critical level of a time horizon. And the way METR does it is they have thresholds for the tasks they care about. So, they have a 50% threshold, meaning if a certain model reaches a 16-hour threshold on this benchmark, that means it can achieve tasks with a 50% success rate that take a human 16 hours. And there is a really rigorous methodology of how they actually measure how it took a human 16 hours, but we will avoid some of those details. The other way we usually think about what long horizon actually means is not with the reference of humans, but instead the reference of models. Some of the relevant model units we usually care about are things like tokens: how many tokens are consumed in a trajectory, how many steps it took, how many tool calls it takes. And these can be really noisy, right? Because I am sure you guys have used different models. A lot of the Codex models are seen as more token efficient than some of the Claude models. And it is a pretty noisy estimate for a couple of reasons. One is that which model you are using, like I just said, and different harnesses you care about have a pretty big impact on how many tokens are actually consumed on a task. So, this can be pretty hard to interpret when you are not holding variables constant. If a task takes a GPT model 500,000 tokens, that doesn't really tell you a lot about what that task would look like for Claude models until you actually run on those Claude models. But despite it being a pretty noisy metric, it is actually really useful and important for us to understand because the amount of tokens that are consumed tells us a lot about how difficult a task actually is for an AI agent to tackle autonomously. We have to deal with things like compaction over long horizons. They don't really stay coherent over enough steps or trajectory length that you achieve. So even though it is a noisy metric, it can be really useful when we look at what a next-generation model can do now, and then you use the same model generation and see if it can achieve a million-token trajectory based on an increased context window or improved compaction. That tells us a lot about how autonomous AI agents can actually go for long periods of time in that sense. And it really defines for us what the technical frontier actually means for models right now. Maybe not really human-adjacent. It is really hard to say how many tokens a task takes for a human because we don't really think in tokens, but it is still very useful in that sense. These are two different approaches we can think about. But what is actually the right way to think about this? The answer is that we probably want to think about all of these. And if we just look at one of these metrics in isolation, it is probably not a great way of measuring things.

CTO: I went through some of the weaknesses with measuring with model-specific metrics like tokens and steps, but there is also a lot of weaknesses in the other approach of relying on humans. What is long horizon for a human isn't necessarily that difficult for a model depending on what the actual task you care about is. There is a lot of tasks that are really tedious and time-intensive. Maybe some financial analyst has to go into an Excel file and fix a bunch of formatting issues throughout the task, maybe changing the theme colors in the file. That might be really tedious for a human to take; it might take them days to do that if it is a really big Excel file. But for a model, it can write a Python script or find some other cool trick to do that really quickly. And that is not really hard for it to do, but you never really expect a financial expert to do that because most of them don't really know how to write these Python scripts. So I think that is one thing to note. And the other that I briefly touched upon before is that the methodology of how we actually measure this has a really big impact. And if someone is out there saying, "Hey, we have some tasks or environments that are 16 hours long on average, someone else is 20 hours," that is really hard to compare across people because there are so many different things in the methodology that really impact what that actually means. It can mean the quality of the experts you are using. Some more experienced experts might actually be way more efficient at doing a certain type of financial or coding task. And this becomes really important as we start shifting towards the frontier of even human capabilities. As you shift towards more long-horizon tasks and tasks that only the top 10%, top 1%, or top 0.1% of humans can really do, these estimates start to get really noisy. And it is something that we really have to consider. The way agents work is developing in its own separate path, and there are a lot of different bottlenecks and different things that AI agents are better at than even the way humans work. And with that in mind, as these paths diverge of how humans do work and what their limitations are and what agents do and what their limitations are, it is really important to keep both these metrics in mind because they paint different pictures of what is actually relevant, and you don't really get the whole picture by just looking at one. So now the question becomes, how do you measure model capabilities? And this is a really important question because fundamentally, long horizon tasks aren't the only thing we care about. This is the larger question that we want to think about every time we are trying to create tasks, create environments to train our models. And the first way we can think about this is environment complexity, and specifically environment complexity related to tool coordination, right? How many tools or external dependencies does the agent have to coordinate? How many tools or external dependencies does the agent have to move information across? Before a long-horizon task world, we will notice that there was a low complexity world where the agent maybe had to read one file or one set of files in a codebase, and that is what a task entailed. But now we can see increasingly as these tasks become more long horizon, what is important to define for measuring model capabilities is that the agent should be using a ton of different tools like Grafana for observability to parse logs, or GitHub for CI/CD, or AWS CloudWatch, or reading and writing to a database. And we are going to notice that as we have these agents use many tools, we also start to think about environment complexity in regards to state changes, which is effectively the degree to which the environment changes throughout the task.

CTO: Fundamentally, all long horizon tasks aren't equal. For example, one task can be made artificially long horizon by chaining together unrelated, independent tasks. However, that doesn't actually meaningfully measure the model capabilities. Instead, a key component of this is actually being able to have the earlier decisions in the environment influence the later decisions. And this comes back to how the agents are asked to interact with the tools, how these tools change the state of the environment, etc. So, we can look at a concrete example. One example where you will see parallelizable complexity, which is effectively not involving a lot of state changes, is when you have an agent analyzing a large codebase and then the agent needs to spawn off multiple sub-agents. It can very easily parallelize this; it can look at a lot of the different files in parallel, come back to the master agent, and wrap this all up. But meanwhile, if we look at sequential complexity, we will see if you have to use a dashboard or logs, a bad early query or a misread can cascade into these downstream steps that really start to have major consequences later on. It is all dependent on how you use those tools and how the state of the environment changed. So the third area that we also need to consider for measuring model capabilities is ambiguity. And ambiguity is defined as the information you give the agent and the environment when starting the task. So this could be the instructions, this could be the artifacts, etc. And increasingly, as these agents work with more artifacts at the start, we want to have them mirror the work that humans really do. And the work that humans really do has a lot to deal with ambiguity. They don't have the most complete information, and they want to let exploration happen. And so we believe that to measure model capabilities, we need to test the model's ability to explore throughout the environment as well and explore these artifacts similar to how a human would. Now the trade-off with this is that if you are going to have ambiguity in the materials you give, there is a lot more possible paths that the agent could take. There is a lot more ways the agent could be right. And that means that standardized evaluation gets much, much harder.

Rayan Garg: Awesome. So I am going to talk about one of the hardest things there are to build in environments and one of the most complex things to really think about where there is a lot of nuance, which is the verifier in the environment. How do we actually know that the work the agent did was correct and give it some reward signal during the training process? So I think there is a few challenges here. Just to give a high-level overview, tasks are getting more complex, the environments are getting more complex, the trajectories are getting longer, and we have shifted a lot. A lot of the early RL that we were doing in recent times was really in hard verifiable domains, and that is why we saw these gains in math and data structure style coding problems. But what has happened over time is now we really care about a bunch of economically valuable work in software domains where we can't just run a Python script or run test cases or write a proof to really see whether or not the output was correct or whether or not the environment was changed correctly. We have to start using other techniques, and the main way we are really going to use that is introduce a judge model or critic model, as some people put it, and they can add a lot of nuance to how we actually determine correctness and assign reward. They will look at two things mainly. One is the final state of the environment and how it was impacted. And the other is looking at the trajectory of how the model that you are training made changes to the state of the environment and what correctness looks like there. Why do we actually use judges and rubrics as a technique? I think it is really important to understand before we can even understand how to use them properly, which there is a few reasons. One is that for these software domains, there is like an entire class of problems that are really important, and a lot of the problems we care about, that you can't really write a deterministic verifier for; they would be really impractical, brittle, or just downright impossible depending on what the problem setup really is. And the other thing also is that not all solutions are really created equal, and not all paths of those solutions are equal either. The worst case of a bad solution we can get is some reward hacking that happens. Lots of different types of reward hacking can happen depending on the setup or the task you care about. An agent can escape a sandbox, maybe see privileged information it shouldn't be seeing about a hidden test suite for a coding task. This is all behavior that we want to prevent, obviously, because those are not actually really valid solutions we care about. And mitigating this is going to require strengthening your verifier and your environment setup, but the judge is really, really important in actually catching this behavior. And that is an important reason why we actually look at the trajectory that the agent actually took to get there.

Rayan Garg: Yeah, and I think there is a lot of careful things you want to be doing here. One is that there is nuance in how much guidance or explicit rigidness you want to add to the trajectories that the model can actually take. If we enforce this too tightly, we collapse the state space of how many paths the agent actually explores. And that can be really bad, especially because some of the more simple approaches we have seen with judges early on is, hey, we will just give it a reference answer or solution or maybe a sample trajectory of what a good solution looks like and then just compare against what the model did and say, hey, does it match up with that? And that really does not work for these more ambiguous or open-ended tasks because there is so many possible correct solutions. It is basically impossible to account for every single one. And we want to check for more robust methods that allow for these different solutions. So now that we have established why we use judges, we want to go through some of the general heuristics and principles we think about when we are designing good judges. Some of the things that we think about at Theta. So you know, I think the first important consideration to make is that judges are agents too. So as environments get really complex, often times a consideration we have is like, hey, we have to make sure the harness can scale and match up with whatever environment you have. Maybe that means introducing a bunch of new tools and making sure your harness can support those tools really well. The agent has clear observability over what is happening in the environment. But I think like we said, the way the judge determines correctness is that it often times has to look at the state of the environment itself as well. So a lot of the harness that you have designed for the agent might also be reused for the judge as well. I think the best way to illustrate this is the example we have here. Let's say you have defined a task where there is some deployment failure with the software engineering task of some platform you are deploying, and the agent's task is to sift through the CI/CD logs on GitHub, look through the CloudWatch logs, figure out whatever happened, apply the changes to the codebase, open a PR, and kick off a redeploy once the PR is merged. For a lot of that, if the judge actually wants to verify whether or not this is correct, besides just looking at the tool calls the agent made—which are usually not very reliable—it actually has to also check the GitHub logs, AWS logs, or the GitHub logs after the deployment happened to make sure, oh, are things actually working properly. So it is really important that the judge has access to the environment in the same way, with some important safeguards of course. One is that we don't want the judge to make an accidental mutation in some way to the environment after the agent is done. So you want to be very careful about that. Maybe that means enforcing read-only permissions for a lot of this information; it can't actually kick off a deployment or anything like that. So those are things to be careful about. But I think this is really important, especially where there is a lot of open-ended approaches and the only way we can really verify correctness is to actually look at the state itself. The answer isn't obvious of whether or not the agent completed the task just from looking at the trajectory. So I think that is one example where this approach is really important. I think the other thing to be notable of is as these environments get more complex, the agent trajectories get longer and longer, and part of the reason we also need the judge to be an agent is that you can't just use this really basic approach of taking the trajectory and stuffing it in the context window of the judge and have it be a basic LLM call. These trajectories can get really long and complex. So we need to do a lot more thoughtful processing of the trajectory in some meaningful way. That might mean we put it into some database. We use sub-agents to actually enrich certain information. Maybe we parse out specific phases that the agent was actually in. Maybe the beginning part was it going through logs. The second part was actually writing code. The third part was actually it checking what happened after that. These are all different things we want to do. And in that sense, we need to make the trajectory itself queryable. So that might mean enriching information like I just said or looking at some other metadata at certain steps. This is really important so the agent can find critical steps like failure points and verify whether or not those are actually failures. Making that usable for the agent is really important.

Rayan Garg: I think another important thing to consider is learnability of our environments. The most important thing here is just the density of the reward signal, and a lot of that comes from your rubric and how the judge is defining that. So, I think you have to be very careful with just overloading with density in your rubric. A lot of times, especially for frontier problems that models aren't really capable of yet, judges will really struggle to apply that rubric consistently. So, there is a lot of QA we need to do to make sure judges are able to apply that information correctly. There is other learnability factors that we care about and we measure in environments like the distribution of tasks and the actual underlying data there. And these are all things we think about for learnability, and it is really important, otherwise you are just wasting a bunch of compute on problems where the model can't actually effectively learn. These are some emerging rubric judge patterns we have seen. I'll quickly skim over this. Often times, deterministic verifiers aren't completely dead. Oftentimes we use them in tandem with judges. Maybe generating an artifact for the judge to look over, where you are collecting metrics. Or another interesting thing we could use is dynamic evaluation-time rubrics, where we are giving partial credit where we have baked in some assumptions that the model made and assume they are correct. It is like grading a test assuming if you got the first part wrong, let's just assume it is correct, did they get the rest of the part right? That can be really important for assigning credit there. I will skip over this part. I will let Ryan just close things off with some things about QA for rubrics.

Rayan Garg: Yep. So for each rubric we produce, we run a couple of different tests. We won't go into all of them. Some of them are pretty basic, right? Gold, no-op variance. These are tests you want to be considering regardless for your verifiers. But I think increasingly, as you involve AI in the process of even creating rubrics or verifying rubrics or aiding experts, you need to have more and more tests, especially as the tasks become more long horizon. And so that really touches on the coverage and the expert agreement. But I think what we wanted to close off with today is why a lot of this stuff matters, right? We spent a lot of time earlier in this presentation defining what long horizon means. And a huge reason we did that is because we feel like a lot of the literature and data being produced right now and being used to train and evaluate models is actually flawed. So we present three major benchmarks in the area of finance predominantly. And so this is GDP-val, Toolbench, and Apex Agents. There's a couple of notable issues here. First, if you look at the average human hours per task, based on what METR has defined for a lot of the leading frontier models, a lot of these different average human hours per task fall far below that, and so they wouldn't actually be considered long horizon tasks. The second notable issue here is that we see that these benchmarks are already reasonably saturated, and we think this is a downstream effect of the average human hours per task. So, it is really important to look at the metrics that are being used here. If you look at the Apex Agents IB section of this benchmark that they put out, pass at 1 effectively means that for 57% of cases, the tasks are 100% solved. That is effectively telling us that there is a large part of these tasks that models are solving similar to what we have seen already. But I think a third key important part here is the breadth. For each of these different benchmarks, particularly GDP-val, they have a very narrow set of Excel tasks that they consider for finance. And for Apex Agents, they're largely focused on IB. What this means is that a lot of these more important areas for learnability like credit, debt, risk in the domain of finance don't really get covered. And then I think lastly, I'll note the reward signal as Gver mentioned is really important. And in regards to the reward signal here, we'll notice that there's like really, you know, if you look at what you need for a rubric, you need a very granular, detailed reward signal. You need, we have 20 different subcriteria and 10 different subcriteria per criteria. So, I think there is a lot of room that is left when you read these benchmarks into how granular reward signal they're giving, which is really important for being able to go ahead and train your models. With that, I think I wanted to round off with a couple of stats about the data we produced. Here you can look at some statistics for our finance data. We can see that the human time to complete one task on average is 15 hours over a 50 task sample set. Furthermore, it takes models a pretty long time to work through these tasks. And after all of that, across all the domains we care about within finance, for example, they still struggle significantly. And so here we provide mean five notably different than all of these previous scores we see here. So, thanks for taking the time to talk with us today. Thank you.