It's inspiring to be on this stage with so many people I admire. My name is Nick, and I'm the Head of AI at Klein. Today, I'll share some lessons we've learned.

Let's start with a bitter truth. For years, we compensated for weak models by building clever scaffolds around them. Ideas like RAG (Retrieval-Augmented Generation), indexing systems, search trees, and tool-calling scaffolds were invented to cope with less capable models. However, frontier models simply bulldoze these abstractions.

Your scaffolding is no longer necessary; it often gets in the way. The key question is shifting from 'How fancy is your agent stack?' to 'How strong is the model driving it?'

This lesson is relentless. A perfect example is Gemini 3.0, released this week. It immediately dominated terminal benchmark leaderboards without any agentic harness supporting it. In this chart, you can see Gemini 3.0 on Terminus scoring better than the vast majority of model-agent combinations out-of-the-box. Remarkably, Terminus is designed as an unopinionated, generic, stripped-down harness with no graph search, RAG, or indexing – just a terminal for the model to figure things out. It crushes. Terminus intentionally lacks clever tool-calling or context engineering features. The takeaway is that capability beats scaffolding. If you get out of the model's way, it will perform well.

So, the key takeaway from this talk is: if you're building agents, relax. Cool it with your clever engineering tricks and stop overthinking it. That's the lesson.

As an aside, talking about clever context tricks and hacks on platforms like Twitter feels a bit played out. I'm tired of seeing it. While it generates engagement, I personally don't find much signal in it.

What I truly want to discuss is the real bottleneck, which receives little attention. Building the cleanest agent doesn't improve model capability by even 1%. Models improve only when labs train on difficult tasks, driven by benchmarks, not agent cleverness or RAG pipelines. Benchmarks determine what frontier models learn next. Models improved at tool use not magically, but because people built RL environments forcing them to practice actions like handling failure and retrying. Agent improvement occurs when models learn in the right environment. Every jump in reasoning and agent reliability has come from a benchmark or an RL environment.

Therefore, the crucial questions are: What is a benchmark? How do you turn real-world agent coding data into an RL environment? What makes a good verifier? How do you detect real difficulty? And how do you train models for problems engineers actually care about? These are the questions for the next frontier.

What is a benchmark? Simply put, it's an environment, like a Docker container, where an agent runs. It includes a starting state (a snapshot of code for a real-world task) and a starting prompt. Finally, a verifier at the end checks if the end state is correct or acceptable.

How are RL environments different? They're not fundamentally different. This chart is similar to the previous one. The key distinction lies in how the reward is used. Benchmarks measure models, while RL environments improve them. The score isn't just for leaderboards; it's used to update the policy model's weights.

How do you transform real-world coding data into useful RL environments for training? At Klein, we created an 'RL Environments Factory.' The first phase involves sub-agents qualifying tasks in parallel to determine their suitability for training environments.

The qualification process starts with validating origins: Does the repository exist? Is the starting commit accessible and open source? It examines the starting prompt and follow-on prompts to understand the user's original intent. Finally, it checks for the outcome: Can we find actual commits or PRs that solved the problem in real life?

We actively look for easy disqualifiers, like 'vibecoded slop' (meaning poorly written or trivial code), or tasks that are too easy, or those lacking reliable start/end states. We don't need more benchmarks for tasks like 'build a Next.js app from scratch'.

What makes a good RL environment and verifier? Phase two builds the RL environment. This involves 'archaeology': reconstructing both states locally, pulling down code, implementing it yourself, and verifying the bug and solution exist. Document all obstacles and dependencies. Containerize it with Docker, removing Git to prevent reward hacking. Finally, define the verifier. This is an art. Consider the analogy of a teakettle: the goal is to boil water.

A good verifier is the whistle attachment. It's a pure outcome verification – the water either boiled or it didn't, whistling or not. The kettle doesn't care *how* it was heated (gas, induction, campfire), only the result. Beware of 'bad tests' emerging from ground truth details, like checking if the burner was set to high or if it was on the front left burner. The key is: don't overprescribe based on ground truth; test for the *spirit* and *outcome* of the task, not the specific steps.

The outcome is a containerized benchmark/environment for the task. Agent work is recorded, showing its trajectory, allowing reliable scoring and verification. It's fully portable. Our path to automation involves fully automating the conversion of real-world coding data into RL training environments.

This process, initially manual and taking 16 hours per task, now takes under 20 minutes. We're building a fully automated RL environment factory, shifting the bottleneck from engineering to collecting high-quality tasks.

An interesting endpoint: what if we built RL environments to test how well agents can *create* RL environments – a meta-benchmark? As models become adept at generating their own training environments from user data, this loop completes. Something to ponder.

Now, for the 'truth nuke,' also known as TRO. It's an unspoken fact: Klein isn't alone in building these systems. Every major agent lab captures this data and does similar work behind the scenes, though they don't discuss it. You know who they are. These companies cite internal benchmarks to justify legacy systems, but you can never study or inspect them because they aren't published openly.

This valuable data, the only thing that truly moves the needle, is not shared. Agent labs stand between real-world engineers and models, holding a unique historical role. We can build better prompts and tools, but that doesn't improve underlying models. We possess the world's richest dataset of real engineering work. Models don't improve without this data, and keeping it closed slows down frontier research.

Today, we're announcing Klein Bench. This is our attempt to create a benchmark that isn't 'cosplay engineering' – not just tasks like 'write a server that generates Fibonacci sequences.' It's real software development captured and packaged into standardized RL and evaluation environments. We always wished someone else would build this benchmark; since no one did, we are.

Anyone can participate. The entire system is open source, with no secret sauce or locked datasets. You can run and inspect it yourself. These environments can be used for SFT, RL, evaluation, etc. The goal is to provide the ecosystem with a real substrate for measuring and improving models, not just code puzzles.

This succeeds only with community contribution. The good news: simply work on your open-source project with the Klein provider enabled and opt into Klein Bench. If a frontier model gets stuck and you fix it, that's an ideal candidate task for a benchmark. Use the Klein provider, identify where models struggle, and we'll incorporate it into this open-source benchmark. Klein Bench will always remain free, fully open source, and accessible. Thank you. If you want to contribute...