OK. Hello everyone. Thank you for coming to this incredibly interesting technical lecture. This lecture has a very long title. So let me briefly summarize the point at the beginning. You write skill files and fill them with lots of instructions. At some point, the model stops tracking them all . The question is, where is this moment? At what point did you add too many instructions to your skill files? And the answer to that has changed significantly over the past year. I'm Lori. I am the Head of Developer Relations at Arize AI. In the past, I was a co-founder of NPM Inc. So some of you may know me from my JavaScript days. Today, I spend a lot of time thinking about AI and how to test it. A few months ago I went to the AI ​​ Engineer conference in Miami, which was great. And I watched Dexter Horthy's performance. It was a good performance. He was n't on this topic at all. But during his speech, he casually mentioned that an agent can follow up to 200 instructions before it starts to forget them. Then he continued his speech, and it was just a brief remark. He also noted that this figure is from 2025, so the situation may be better now. I stopped listening for a second because I thought: 200 instructions is not a lot, right? A decent skills file instantly exceeds these 200 instructions. If a user says “ do X, then Y,” “ always add section Z,” “never use phrase W” —each of these requirements is a separate instruction. And if the model quietly stops tracking them after 200, that's the real limit to the complexity of what you can create . So I wanted to know where he got that number from and if it was true. You know that feeling I'm talking about. You write a big, wonderful skills file: rules pages, edge cases, tone, formatting. You hand it over to an agent, they do the work, and you look at the result and think, “Did he really pay attention?” “Did he really follow all these rules, or did he just do what he wanted and produce something similar to what I expected?” You can't know for sure, can you? More on that later. So you live with a constant mild anxiety every time you press "start," and that's what this study is about, and we're trying to figure out if we can avoid that. Here is my promise for the next 18 minutes. I'll show you where that number 200 came from, whether it's still relevant, and what the real number is today, as it has changed by an order of magnitude. And then we'll talk about what that means for you. How long your tasks and requests can actually be, and what changes to your workflow you should make as a result.

So, the number 200 is not a myth. It comes from a real-world test called IFEval from a paper written by Yaroslav Wich and co-authors last year. And this test is surprisingly simple. This is how IFEval works. You ask the model to write a business report and give her a list of specific words that she must include in the report. Include the exact word "client." Include the exact word "income" and so on for as many words as you want. Each of them is an instruction that she must follow. And then you count how many of those exact words appeared. Because the test is so simple, you only need to keep two numbers in mind. The first is the density, which we call N. This is the number of rules that are being discussed at the same time. And the second is accuracy, that is, the percentage of rules that it was actually able to execute. You might say that including random words in a report is not the same as following real instructions, and that's fair, we'll talk about that . But keywords are just a proxy indicator. “Include the word revenue” is the same type of task as “ include a pricing section,” right? Or "never use that phrase." This is a separate, clearly defined restriction that you place on the agent, and they must adhere to it. If a model can't track 200 words in a single query, it will definitely have difficulty with 200 more complex instructions. Therefore, most likely, she will cope even worse. So, this number is an upper limit. This number is the maximum that can be achieved. If you give more complex instructions, this number will likely become smaller, and 200 is a very low limit. So before chasing new models, we need to do some quality science, which means reproducing the old result and making sure that the 200 limit is realistic. So, I reran the original benchmark. The original article tested a whole range of models, and models live and die very quickly. So by the time I undertook this testing, only three models from the initial set used by the authors remained available through the API. These were GPT-4.1, Claude Sonnet 4, and Gemini 2.5 Pro. These were models available 12 months ago that remain available now. And that's why we protested these three, because that's all that's left. And since I first published this research a few weeks ago, one of those three models has been retired . So this was the last moment I could take this test. So, we only have two left from that list. So don't get attached to your models. Here are the results we got reproducing the original F- scale findings. This is the accuracy on the vertical axis. It starts at 100% and starts to fall. and the number of rules increases along the bottom axis on a logarithmic scale. So every time we get halfway through , the number of rules the model works with doubles. So, on 500 rules, you lose 30, 40, 50% of them . Our curves matched the results of the original article within the margin of error, so the conclusion was realistic. A year ago, somewhere around 200–300 rules, advanced models started to fail. This is a very low ceiling. So, that's our baseline, and now the fun part: we took the same test and applied it to today's cutting-edge models, or whatever was cutting-edge at the time of the test. I tested GPT-5.5, Claude Opus 4.7 (because 4.8 came out a week after my test), Gemini 3.1 Pro, and DeepSeek V4 Pro. I gave them the same prompt, the same words, the same everything , and immediately ran into a problem: they did a great job. They all immediately got 100% on this test. No mistakes. So, we created a test to find a ceiling, and the models went through that ceiling without even noticing it. And that was a problem, because the benchmark was designed for a maximum of 500 words. So I had to change the benchmark to find a new ceiling. So I moved the targets. I gave them more words to include. I doubled it from 500 to 1,000. I doubled it again from 1,000 to 2,000. And I kept doing that until I reached a vocabulary of 10,000 words, and that's where I started to find the limit of what modern models are capable of. Um, so let me show you this. This is the main slide. Here are the results. Remember that the X-axis is a logarithmic scale. So it goes from 500 to 1,000, 5,000, and 10,000. It looks like this scale drops off dramatically, and it actually does at around a thousand values. Ahem. But look how far to the right these new curves go before they start to bend. A year ago they “broke” at 200–300 instructions, and now, depending on the model, the limit is closer to 2,000, and for the best of them, up to 5,000 instructions, before they start to lose ground sharply. So , in about 12 months, advanced models became almost 10 times better at executing instructions simultaneously . This is the main conclusion, but there are many nuances that we should delve into. The ability to track 2000 named constraints in a single query already exists. And it's really interesting because I think—I don't know if others feel this way, but it seemed to me that the jump from, say, GPT 5.1 to GPT 5.5 was pretty gradual, right? It didn't feel like we were 10 times better, but it's a test that's really important for practical things like: how long can my skills file be ? And in a year we became 10 times better. And what amazes me is that this benchmark is barely a year old. A year later, 500 is already a rounding error, and all this continues to change right under my feet. I tested 4.7 Opus, 4.7 Opus 4.8 is even better. So this diagram is a bit outdated, which is actually the main point . If you have defined your engineering assumptions about how skill files should work, how long your queries can be, and you did it over 6 months ago, you are wrong now and you should probably reconsider your approach.

But there's more to this story, because the way models fail has changed dramatically , and exactly how they go wrong is very important. This part was a completely unexpected discovery when I started running the experiment, and at first it completely ruined my test because the old crash mode was boring. They simply forgot the instructions, and I could measure how many instructions they remembered and how many they forgot. But new models break in their own strange, completely characteristic way. So let me give you an idea of exactly how these four models fail. Deep Seek 4 is a traditional model. She just forgets things. There is no drama with her. She starts forgetting instructions after about 750 rules, and by 2000 she loses almost half of them. So she just forgets, which is frankly the type of failure I trust the most because it's predictable. This is very easy to measure. And other models were far from being as convenient to use. Opus 4.7 consistently decided that the test was unsafe. And it simply refused to run the API level test. I didn't know there was this API response from Claude that seemed to say, "No, I could do that, but I won't." But this is a fully supported answer at the API level, because they care a lot about security. And I started receiving such messages all the time. The reason is that Claude has a very sensitive security classifier . If you add certain combinations of words, such as anthrax and cyanide, it decides that the entire query is dangerous and stops. If you remember how my test works, it consists of throwing 5 to 10 thousand random words into an instruction file. So , my randomly chosen words contained all sorts of things that looked dangerous in combination to the security filter. So he kept refusing , claiming that I was asking him to, you know, build a bomb or something like that. So, to get Claude to cooperate, I had to run all my words through OpenAI's security filter and remove all the " suspicious" words so he could do anything. Having received this , Claude did very well. But the peculiarity of the glitch is that Claude more often decides that you are doing something dangerous, at a very early stage. Even on two or three hundred instructions, if what you're doing contains anything related to medical advice, because medical terms often have a double meaning . They can be both dangerous and safe. So, the third type of failure was in Gemini 3.1 Pro. Gemini runs stably up to 5000 instructions. It shows extremely good results. Overall one of the best on the chart. And then something strange begins. He doesn't forget instructions. He is simply overwhelmed by them. He tries to use reflection tokens to make sure he follows all the instructions at once . And when the number of instructions becomes too large, it uses all its tokens for thinking. He spends his entire token budget on thinking and then doesn't produce any answers. That is, if you give him a task for 10,000 tokens, he will spend 9,500 on thinking, and then give a 500-word answer that will not contain any of the required tokens. So he's cornering himself and not having enough space to respond, which is very expensive and completely ineffective, although that's totally his style, right? Although, you know, I would never say that out loud. And finally, the winner is GPT-5.5. GPT-5.5 is the best of all, 99% accuracy even with 5000 rules. But if you push him hard enough, he becomes the strangest of all, because he doesn't refuse outright and he doesn't forget silently. Instead, he just gets frustrated and says the test is pointless. He starts the report, you know, that's the thing, he doesn't just say "no" right away. He starts writing a report, and after about 500 words he's like, "No, that's stupid." "I won't do that." And then he politely says, "That's ridiculous." "I won't do this anymore. " This is the actual answer he gave me, but it was somewhere around word 5,000 of a business report I had him create. So he's right, right? I asked for a meaningful business report without a specific topic, containing 5,000 random words. You're right, Gemini GPT. That was a stupid demand. This is a deeply unreasonable request, and GPT has pointed this out . But this is still considered a test failure because the unfinished report it produces is missing most of the keywords, and this is the hardest to detect because Claude gives up right away. Claude says, "No , I won't do that." Well, Deep Seek tries its best, uh , but GPT does what looks like a good job until you read the report to the end, where it says, " No, actually I'm going to leave because this is stupid." Um, so if you look at them all together, Deep Seek quietly forgets, Claude gets scared and gives up, Gemini thinks too much and shuts up, and GPT-5.5 does half the work and tells you the rest is too primitive for it. Um, and the point is... the point is not which one of them is the funniest, although that is indeed a bit funny. Well, the point is that the question "did he follow my instructions" no longer has one option for failure. There are four different ways to fail, and you won't be able to recognize this failure unless you know which model you're dealing with and what its failure behavior pattern will be .

Well, so the models have become 10 times better. They fail in funny ways. Why should this worry you when you get back to your desk? Because three things have changed in your workflow. The first is that a year ago it was smart to keep the skills file very, very short. Uh, up to 200 instructions, then links to sub-skills and a whole, you know, Byzantine maze of, uh, additional skill files, sub-files, and the like. Oh, and you were squeezing your instructions to fit into the very small space available, but you don't have to do that anymore. Your skill files can be very long. Um, number two—is that if your case requires 100 or 300 specific rules, you can just put them all in the query. Well, you don't have to lie awake wondering which ones the model silently ignored. Um, and if you've been thinking, uh, about your own experience using models , uh, you'll probably recognize this. You found that you started to worry less about how long your query was going to be , uh, because the models actually became 10 times better at following your prompts. Um, 2000 named restrictions— that’s a whole style guide, right? It's like every brand rule, every legal disclaimer. Well, a year ago you would have had to distribute this among a dozen specialized agents and hope that your specialized agents clearly delegated tasks to each other, but now you can ignore that. Um, but the third thing is the most important. The question used to be: can a model even do this? And the answer is now firm: yes. Well, it's solid enough. Well, is it worth the cost? That's a new question. Because you can include 10,000 words, sorry, 10,000 different instructions in your query, but it will be a huge query. This will be a very expensive request. This will be a very slow request. So what used to be a blank wall you were running into is now a flexible choice: Should I add all these extra instructions if it will result in higher costs and more latency? Well, now a few caveats to get ahead of the questions . Um, first and foremost, I mentioned this before, is the proxy task: including random words in a fake business report is evidence that the long skills file is working. This is not the same as proof that a long skills file works. Um, also the models hit the wall at very different points, from 750 to over 9,000. So you need to choose your model very carefully. Well, our test doesn't measure whether the model clearly reasoned over a giant query. So the good news is that since I did my research a few weeks ago, a whole bunch of people have gotten involved . Um, and now there's some good research. Real scientists got involved, and Chromatic conducted a “ context drought” study on 18 models, showing that accuracy on long inputs can drop by 30–50% long before the context window limit is reached. Well , the most surprising part of their discovery was that coherent, well-structured text is more likely to encounter this failure mode than if you just randomly arrange the instructions and mix them up. Well, I do n't know why this happens. I should read the report. Um, so the model can track 2,000, 5,000, maybe 10,000 instructions, but it won't necessarily be able to reason clearly about them. If these instructions contradict each other, if there is tension between them , the model will not necessarily handle it correctly. Um, and there's something else I mentioned in passing. Well, Claude's refusals are annoying, but they are loud. You get an error and know that something went wrong. Eh, a polite, underwritten GPT report is much more dangerous because it looks like a real answer. Well , you have to read all this to notice that he quietly gave up halfway through, which means you can't trust the outcome. This means you have to re-read the result every time to make sure it works or not. So, the model will take your 2,000 rules and give you something that at first glance seems confident and polished, but may break down halfway through. By the way, people always ask me how much did all this cost me? Completing all these requests cost me $209. 2,300 calls to seven models cost $209. It turns out that the latest research is n't that expensive. Ahem. This is the part of the conversation where I say you should test this in a production environment because you can't be sure your model won't fail silently. You knew I would mention evaluation because I work at A Rise and that's what I do there. But I won't abuse the A Rise advertising. I'll tell you one truth: if you build a real AI application and give it really complex tasks, you will encounter one or more of these advanced failure modes. And if Claude at the API level sends you to hell, the only way to know that something went wrong is to monitor the output data with another LLM. This is what evaluation is, this is what A Rise does, and this is where I will stop. I already mentioned that new ones have emerged since our research . Here's another important study, a paper testing 46 models called " Reviewing the Reliability of Language Models in Executing Instructions, " which, you can be sure, caught my attention after my own work. They discovered something unpleasant: a model can successfully pass a test like ours and still remain extremely unreliable. Because if you rephrase the same instruction slightly differently, it can drastically change the quality of its execution. A model can execute 2,000 instructions very well, but if you change the order of those same 2,000 instructions, it can suddenly start executing them much worse. And how exactly to do this? What order of instructions to give to a model so that it executes them perfectly and doesn't get confused is a question that is still being researched. So, capabilities have increased, but reliability remains a problem. And that's just a little bit of bragging, because I was happy about it. For example, I am not a scientist. I did the research, and then a whole bunch of real scientists got involved and did real scientific work on the same question. Now there are a whole host of tests that have emerged to measure the same question. Fire bench , CCR bench, guide bench— they all try to measure the same thing. How well the models adhere to many real-world, complex constraints simultaneously. And now the entire industry is paying attention to this. So if you need better science than my, you know , 10,000 random words, then real science already exists. So that brings me to where I'll leave you. A year ago, the hardest part about writing a skill was fitting everything in so that the model didn't lose its essence. It was a compression problem, and that compression problem is no longer there. The model will handle your 2000 instructions just fine. The new difficulty is knowing whether she did what you said , and that's the problem of verification. The verification problem is not solved by writing a better prompt. It is solved by checking the result each time, just like you test any other code, i.e. through evaluation. The ceiling has risen 10 times in 1 year. So go back and check the assumptions you made 6 months ago about how big your prompt should be, how big your instructions should be, because they may already be wrong. So, that's all I wanted to say. If you need all the code and data, it's at this link on GitHub. And this other QR code is what the marketers forced me to insert . We're having a World Cup viewing party today at 5:00 PM . You can come to our party. This is a link to Luma that will help you get to the party. I hope this conversation has given you new information or at least made you laugh, and thank you very much for your time and attention.