Kobe Crawford: This is the the last presentation I have to give this uh conference. So, I'm feeling already a little bit of the euphoria of like, "Ah, it's all done." Um I know we're we're at a uh at a uh also closer to the end of the whole sequence uh is hasn't been I mean I I keep finding these conferences to be like some of the highest signal that I get like wherever I'm wherever I go. So, how how generally is it people feel like like they're getting what they came for here? I'm just curious because like I you know we're we're Snorkel um when we we put in a sponsorship and we want to know that like people are getting what they want out of they know they're going to come back because we want to sponsor next time. We want to know people are happy about it. So, did did did you did you guys see what you wanted to see? Yeah?

Audience: Yeah. Yeah, really good. Brilliant.

Kobe Crawford: Brilliant. Um so, now that it is 3:45, I'm going to go ahead and start the the official thing. Uh so, uh my name is Kobe Crawford. Uh I'm a developer advocate at Snorkel. Uh we call ourselves the Frontier Data Frontier AI Data Lab. And the what we're doing right now our main thing is starting from the research-backed work that Snorkel's been doing since its inception. Uh we've been working on a variety of things about data quality and then then at this point now where where we're focused is actually providing data sets where we assure a certain level of quality. We're very attentive to being very very very motivated about making sure the data is high quality. And part of how we get to high quality is we always make sure to have sort of an expert in the loop as part of the process. So, we have uh expert contributors that we work with and we bring people in uh to provide their expertise to make sure that the data that we generate uh is of top quality. And then for the for the top labs that want to use our data to uh improve their models and get them get the hill climbing done uh in the right way, uh that's what we that's what we do at Snorkel.

Kobe Crawford: Um because of that, um you know, uh a lot of what goes on is uh still more research. And uh this is a a talk that's talking about some of the work that we did that our research team did. And um one of the keys in this research is uh that we're looking at uh how the best quality data can be best applied and like where it is that we need to be looking for where there are opportunities uh to get that done. So, in this particular case, talking about stop making models bigger. I mean, it's a nice punchy title. Of course, we don't really mean like models shouldn't be large intrinsically, but uh the point broadly speaking is that sometimes we find great wins to be had with the right data applied to the right uh to the right problem statement. So, this is something we're going to talk about a specific use case that we that our research team discovered and uh and in partnership with the RLLM team, uh which is a research group uh with a part of UC Berkeley. Uh and so, the UC Berkeley team over there, RLLM, the Agentic Project, their their lab partnered with us on this particular work.

Kobe Crawford: So, the goal this is the the as I said is making a 4 billion parameter model outperform outperform a 235 billion parameter model on a tool use task for financial analysis. So, we'll start with the research objective. And then we'll iterate in through talking about the approach that we that was used for this particular process and then talk about the results and um happy to report that we got what we were looking for. So, uh so so, good things to be had. So, couple of quick level setting backgrounds of the what we're talking about here. First is that as we see enterprise use cases uh take on some greater complexity, we have obviously we've got the massive explosion with people are doing in terms of personal assistance. And as people are working in the context of enterprise, a lot of times you still need a several more constrained uh choice about how to implement something and make sure that it's reliable. And like when you're looking for things that are going to be done uh for uh you know, enterprise production use cases, you kind of almost have to make sure there's a lot of safety and security things done.

Kobe Crawford: So, these other kind of priorities and that that fold into what people typically want to do, we're looking at these things and saying, "Okay, well, these are the enterprise use cases that people have." And as people try to solve the problems of making the models perform at the level that makes it like acceptable for actually being deployed as a production service, um we see very often that people choose like, "Well, okay, we didn't get the performance that we wanted with this right now. We'll just drop in a larger model. It'll be smarter. It has greater reasoning skills. And we'll just sort of expect that the important performance will improve uh commensurate with the additional load of the size of the model and the the the greater inference cost that goes along with that." And in some cases that might not always be the right thing. So, we see people saying, "You know, we'll just get a bigger model. That'll solve the problem." And uh sometimes maybe that isn't quite the answer. In this case, what we're trying to do is to say, "Can't we take a smaller model and then use RL with the right data to yield the kind of performance gains that we're looking for and to deliver the kind of application functionality that we want?" And so, that's the target here.

Kobe Crawford: Uh and again, for these various reasons, cost, speed, security, and then the the idea that in general, you know, you start with a really big model and make your POC and make it work and everybody's happy that it works. It's like, "Okay, now what do we do to productionize it?" You want to roll to production, you want to think about how you're able to deploy that. Do you need to keep everything on premise? You do do you have the ability to deploy and run your service yourself so that you don't have to have external dependencies and worry about the data export aspects and data control. Especially in the context of financial data and health care and other domains like that, people have to be concerned about those aspects as well. So, for getting a smaller model to be able to perform as well as larger models, uh we feel like in the particular case of talking about tool use for financial analysis that RL is the right time to to be uh making the kind of training. You're talking about like changing the behavior. And so, that's kind of more of a behavior thing. The RL's kind of better for behavior than say you're talking about like changing the core data and knowledge that's inside of that. So, that's that's an intuition about like how we've approached it and that's part of what's going on here.

Kobe Crawford: So, a larger model, sometimes it's more like uh taking a sledgehammer to crack a walnut. It's like just adding all of this capability is like this. And the RLLM team uh that we worked with, they talked about this and their description of it was the Terence Tao effect. Uh uh Terence Tao, the famous mathematician who's uh uh I forget what awards he's won and whatnot, but well known for being, you know, generally brilliant about mathematics across the board. And therefore, like could approach and manage any kind of mathematical problem. But that much brilliance might not necessarily be what a financial analyst actually has to have. They don't have to know all the kinds of math. They don't have to do latent Dirichlet algorithm stuff uh to talk about, you know, doing a SQL query and getting some math getting some some data's back and then, you know, doing some addition and subtraction, right? So, the the idea that you must always get to a much smarter model to do something or deeper reasoning to get something done well uh is the thing we're challenging here.

Kobe Crawford: Um so, here is that 235 billion quant three model responding to the question in this environment that we built. I'm going to talk about the environment a little bit more in detail later. But I point this out to sort of show here's a reasoning model, a smarter model, and its response in the context of needing to actually use tools. So, the response that it got that it generated to the question, "What is the year-over-year growth rate of YouTube ads revenue from '23 to '24?" began with uh making a query to find an existing uh some values, but they the query it chose was to a non-existent table. The table didn't exist. It didn't actually inspect the environment and inspect the tools to find out what tables it could query. It just threw a query out uh without doing that. Um so, the table it wasn't there and didn't get anything back. It guesses again. Still doesn't get anything back. And then having not gotten anything back in either of those two uh attempts, it falls back to just hallucinating an answer. And so, out comes this hallucinated answer. It's completely, you know, don't know what the weights told it to say, but that what came out. And, you know, it's it's not very useful.

Kobe Crawford: So, even though the model is incredible in terms of like much better at reasoning than a much smaller model would be, uh that greater reasoning did not help it when it needed to use the tools. We're going to come back to the same question again against the model that we fine-tuned that's only 4 billion parameters. And you're going to see the difference. We'll talk a little bit more about those differences later. So, put a pin in that. Come back. We'll see that that year-over-year question from come back. So, here, this is what we're talking about. Summarizing it again, no discipline in tool use even though it has all the the abilities to reason that it has.

Kobe Crawford: Moving forward to then what we did for this uh attempt to use RL to make the smaller model work well. Uh the first thing is to generate a high-quality data set. Um at Snorkel, our general approach is again to have experts in the loop. I don't know if I said I say again now, but I don't think I said it. We have experts in the loop for the data that we do. Um the way we generate data and the way we work on it is we have a platform that we've used internally for interacting with things we we solicit the work and support of experts on various tasks and various topics. So, if we need somebody who is in the financial analysis space already, then we get them and pull them in. We'll work with people at the PhD level for the their domains of expertise. Uh and also, of course, people who are deep in the industry and have been working for some time and then know the space well.

Kobe Crawford: Um The process of doing that that's like one of the things that we put an emphasis on is how we work at Snorkel for our data generation. And then broadly speaking, naturally that can be augmented with other kinds of things. But that's what we're really key about like what we want to do in terms of emphasizing quality as a core element. So we have the data set. And then we go through and make sure there's a verification step done to make sure that the tasks that are defined from that data set are actually appropriately fit fit it to the task and are actually like good tasks that they do in terms of like the it can be queried. You know, you you know that you're going to get the results that you need from it. And that that we should be able to have a verifiable answer that we're looking for. So we do all the verification steps to make sure that everything is correct on that front. And that's another part of what means to you know, put together the data set and have it ready for use. Data quality again is a big emphasis for us. So we really make sure that's a the key.

Kobe Crawford: And then it was time to do our RL with it. And the way we did our the way that this was done we're talking about very few surprises in terms of like you've seen the state of the art in this in this space. GRPO Again, we started with a 4 billion parameter model. And then the environment that we used the RLLM framework again through the UC Berkeley partnership through the the developers of that framework. And we have our a FinQA environment that we built. And we're going to talk a little bit more about the details about that environment in just a moment. But then this is something that was able to be done like in a 24 21-hour job and the total cost of running that job was under $500 per run. So um RL is not have to be a very expensive thing to be able to get non-trivial performance gains. And if you're already working about working on working with models that you want to host yourself, if you're already thinking about like what you'd like to do to be able to do things we have on prem kind of solutions or things where you're doing it with a smaller models. And you aren't already thinking that you can improve the models the way you that you want, then this is like a call to action that you actually can. That it's actually a very tractable thing to get a model that you want to work with actually up to the performance levels that you need using RL. Even if Karpathy doesn't like it.

Kobe Crawford: Um So our FinQA environment is something that we built. It's set up for being able to host the kinds of questions that are being done here. It provides a specific set of tools. It's set up where like everything is built into the environment. So there's no external dependencies that are that might be you know, in some remote data center that you don't have access to. So when you deploy the environment it's fully self-contained. Kind of roll out that if you worked with something like Harbor before or if you worked with like OpenEnv, you're familiar with the same thing about using an environment like this. And this is an environment that we've actually built and published. It's available on PrimeIntellect's infrastructure as well. You can so somebody can load up right there at PrimeIntellect. Also on OpenEnv and and and actually saved into the OpenEnv repo on GitHub. And then the OpenEnv the PyTorch folks and and Hugging Face folks team up and host these in Hugging Face spaces. So these kinds of things are accessible and easy to find if you want to take a look at them and see how you might take take them apply them to your needs. And then again like getting getting started with RL is actually easier and easier these days.

Kobe Crawford: We have the FinQA set up where we have 290 samples that way and we have our more advanced 79 samples called FinQA reasoning that requires multi-table queries. And so there's enough enough of the reasoning that has to be done across that to make that we've we've identified that these are harder tasks. And so we have like essentially two benchmarks that are built inside of this environment. So that's the set up of like how we get this done. We're going to go about talking about the evals and the results that we got working with this now given the RL that we just did on this 4 billion parameter model.

Kobe Crawford: So we did it. It's performs better than the 235 billion parameter model with that RL training loop. And the performance in terms of pass at one was essentially double of what it had been percentage-wise in terms of like solving problems. So it's a very significant uplift that was done with this $500 loop. And again, the right data set and and is really a key. You want to get the questions and answers to be actually things that are really going to help the model learn. But what is also interesting is what was important about what the model needed to learn.

Kobe Crawford: So just to give you a little flavor of like what that 4 billion parameter model looks like in terms of like how it behaves. And if you recall what we talked about earlier, the 235 billion parameter model tried some queries without knowing what the tables were. Didn't find anything and then hallucinated an answer. This 4 billion parameter model having been fine-tuned on this data set tries a table and actually first discovers the tables by using the tool get table names. The tool existed for the other model as well and it just didn't choose to try it. So the first thing it did was actually query to find out what tables it had available to it. So that's already like win. All right. The the second thing is then from there it went on to actually inspect the schema. Let me find out what's in that table so I know how to make the right SQL query. And so it's like does get table info to get the information back to know what to query it.

Kobe Crawford: Following that it runs a query. Actually ran into an error. It actually asked for the revenue column. But that column was not actually a part of the the data in the table. Given that error it actually corrected. It self-corrected. It observed the error, responded to that error by actually correcting to find the actual column that it needed. And so you're seeing both the error correction that it had learned how to do as well as the use of the tools to discover the right information in the first place. So between those two, those behaviors are the real keys to succeeding at these questions. And this is actually something like maybe not quite intuitive about like where it is that the model was failing. The reality is that what it needed to do and here it is getting the correct answer. The reality is what it needed to do was to learn how to use tools.

Kobe Crawford: Couple interesting things that go along with it that are more more fun and also really useful and good for our situation here. The training data that we talked about at the beginning, there were single-table questions, multi-table questions included in the overall data set. And for as part of the evaluation study, one of the things that they said was like let's take a look and see if we train with single table only, train with multi-table mixed in so the full data set across both types, or try to do some curriculum learning and actually start with single table, let the model kind of climb a bit and then progressively add multi-table. And it turned out the single-table only training was actually the one that yielded the greatest uplift for these kinds of questions. So that was a nice pleasant surprise.

Kobe Crawford: And the other surprising thing was that even though the single-table only training regime was the best training regime the uplift that we see in terms of the model's performance on that harder benchmark that is a that has multi-table questions was a similar doubling in in in percentage improvement. So the harder multi-question multi-table Q&A in the FinQA reasoning question set also saw 13.9 to 26.6 percentage jump after this training. So interestingly enough again, the tool discipline, just knowing how to use the tools that are in the environment turned out to be a bigger deal than anything else in terms of how to make these models actually get better at what they need to do in this space. So turned out it wasn't the reasoning that was the issue. It was the tool use. We focused only on single step for the best performance. And we're able to fix that core failure mode. And given that core failure mode being fixed, it turned out that that then made the model better in terms of like the improvement generalizing to other question sets.

Kobe Crawford: And so that means like that's the key to talk to take away from this is that sometimes the the idea is to find the specific behavior that's really the problem. And one of the things that I would go back to what we do at Snorkel, one of the one of the things that our research team has been talking about a lot lately is building rubrics as part of our evals. And then those rubrics by breaking down the rightness or wrongness of a model's response into a full list of different questions that can be answered. And looking at each of those individual questions, you can then start to use the rubric as as a way to find and intuit like and find where the actual problem is among all the multiple possible arenas. So instead of simply knowing yes or no at the final, which is good for the RL part you can use the rubric to help you do an analysis of what are the behaviors that you want to actually generate data sets to help you with. So you make decisions about what which data sets you need or which data you want to work with based on what you see coming out of the richer feedback that the rubric gives you. And then the RL still gets a single value or so GRPO just usually works with a single value. That's it. That's part of how it works. So you use that for the actual RL cycle.

Kobe Crawford: So that's the that's the summary of what we we did with that. We think it's a really interesting result to know. And you know, again the opportunity of what you can do with the solving the right questions or the right problems really helps. This link here is to a blog post that we have about this. So if you have questions about the details of this particular study and you want to see more about it, you can drill down within that. It also links to a partner post from the Agentyca team over UC Berkeley. So their post also has additional information you can see from them and and this is you know, the significant thing we wanted to talk about. So thank you for your time and I I don't know how much time we have left for questions or not. How are we doing? Are we already at time? Looks like it. Yeah, sorry. Okay, so I'm sorry we don't have questions. I'll