About a month ago, we at Cerebrus and OpenAI released a new model, state-of-the-art model called Codex Spark. Codex Spark can generate code at 1,200 tokens per second. To put that into perspective, if you look at the Sonnet family or the Opus family, those can generate code at about 40 to 60 tokens per second. In this new era, as we're starting to see much faster coding models, this is 20 times faster. Not only does it unlock new capabilities and use cases, but it also requires us to rethink how we as developers interact with the coding model.

A lot of these bad habits that we had before that were generating maybe 50 tokens per second of bad code, unless we fix them, they're going to start generating 1,200 tokens per second of bad code.

One of the biggest things that we have to think about with hardware is the memory wall... And so when we are running inference, we have to constantly move our weights and KV cache values between memory and our actual chip.

What a lot of newer companies are doing are thinking about companies like Cerebrus or Groq, they're thinking about how do we move this memory to be as close to the chip as possible. And so here's an example of the Cerebrus wafer where all of the chip... all the memory is distributed across the chip in SRAM.

Disaggregated inference really has become commercialized in the last few months. In traditional inference, there's two steps: prefill and decode. Prefill is where we're taking every token that the user inputs and processing it, embedding it, and adding it to our KV cache. This is a step that can happen in parallel and so it's compute-bound. Decode on the other hand is where we're actually generating the output token by token and this is sequential and is as we mentioned memory-bound.

What we're doing and seeing now commercially is that we're splitting up these two steps so that prefill is done on one type of hardware that is compute-optimized and decode is done on another piece of hardware that is memory-optimized.

There's so many ways that we are training our models and shaping our models to cater to our hardware. A great example is a very standard model architecture: mixture of experts. Instead of activating the entire model all at once for every single token, we're only activating a subset of experts for every time.

Inference optimizations... one of the biggest things that we're thinking about at this level is KV cache reuse. And so by storing and reusing previously computed token representations, we don't have to recalculate attention over the sequence at every step.

Another really helpful trick is to actually make skills out of successful sessions and capture trajectories that are working really well.

At 1,200 tokens per second a model like Codex Spark makes validation basically free. There is no excuse and no reason why you should not be doing things like test suites, linting, pre-commit hooks, diff reviews, browser-based QA automations.

So let's say that I want to code a navbar and I want it to be midnight blue. I want four different icons... what I can do with Codex Spark... I can tell it to generate 15 versions... I can cherrypick the version that I like the best.

Now that the models are so fast, it should not be you spawn a session, you go get a hamburger, you scroll Twitter, and then you come back. Now you can actually sit down and it's a real-time collaboration that you're able to have with this model.

As I was mentioning before, it really shouldn't be, you know, you spawn 10 agents, you never verify the code. You don't know what's happening under the scene... You can be super specific... ban the model from deleting files, give it a max diff size, have the model only read and write.

You can just bake this into your automatic workflow so that after every single task on that checklist is complete, you're just asking the model to automatically delete unused imports, clean up unnecessary lines of code, make it so that all of my functions are structured the same way.

A general very high level framework is just always always break up large tasks into smaller bounded goals.

An example of how you can do this and set up an external memory system that is persistent every time you set up a new session is with this four file system. We have agents.md... plan.md... progress.md... verify.md.