SallyAnn DeLucia: Hey everyone, gonna get started here. Thanks so much for joining us today. I'm Sally. I'm the director of RISE. I'm going to be walking you through some of crowd prompt learning. We're actually going to be building a driven optimization loop for the part of the workshop. I come from a technical background and started off in data science before I made my way over to product. I do like to still be touching code today. I think one of my favorite projects that I work on is building our own agent Alex into our platform. So I'm very familiar with all of the pain points and how important it is to optimize your prompt. So I'm going to spend a little bit time on slides. I like to just set the scene, make sure everybody here has context on what we're going to be doing and then we'll jump into the code with me. So, I'll let you do a little bit of an intro.

Fuad Ali: Yeah, thank you so much, Ellen. Great to meet all of you. Excited to be walking through prompt learning with you all. I don't know if you got a chance to see a harness talk yesterday, but hopefully that gave you some good background on how powerful prompting and prompt learning can be. So my name is I'm a product manager here at Arise as well. And like Sally said, we like to stay in code. We'll be doing a few slides, then we'll walk through the code and we'll be floating around helping you guys debug and things like that. My background is also technical. So, I was a backend distributed systems engineer for a long time. So, no stranger to how important observability infrastructure really is. And I think it's an appropriate setting in AWS for that. So, yeah, excited to dive deep into front loading with you all. Thank you.

SallyAnn DeLucia: Awesome. All right, so we're gonna get started. Just give you a little bit of an agenda of the things I'm going to be covering. So we're gonna talk about why agents fail today. what is evening prom learning? I want to go through a case study kind of show youall why this actually works. And we'll talk about learning versus GA. I think everybody I had a few people come up to me over the conference about what about GEA? We have some benchmarking against that and then we'll hop into our workshop. But with this I want to ask a question. How many people here are building agents today?

Audience: Okay, that's what I expected. And how many people actually feel like the agents they're building are reliable?

SallyAnn DeLucia: Yeah, that's what I also thought. So let's talk a little bit about why agents fail today. So why do they fail? Well, there's a few things that we're seeing with a lot of our folks and we're seeing even internally as we build with Alex for why agents are breaking. So I think that a lot of times it's not because the models are weak. It's a lot of times the environment and the instructions are weak. So having no instructions from their learned environment, no planning or very static planning. I feel like a lot of agents right now don't have planning. We do have some good examples of planning like we have Claude Code, Cursor. Those are really great examples but I'm not seeing it make its way into every agent that I come across. Missing tools big one. Sometimes you just don't have the tool sets that you need. And then missing tool guidance on which of the tools we should be picking and then context engineering continues to be a big struggle for folks. If I were to distill this out, I think it's these three core issues. So adaptability and selfarning. So no system instructions learned from the environment touched on determinism versus non-determinism balance. So having the planning or no planning versus doing a very static planning. You want to have some flexibility there. And then context engineering I think is a term that just emerged in the last six to eight months but it's something that's really important that we're finding missing tools tool guidance just not having context or confirming your data and not giving the LM enough context. So these are the core issues to still. But I think there's one other pretty important thing. And that is this distribution of who's responsible for what. So there's these technical users, your AI engineers, your data scientists, developers, and they're really responsible for the code automation pipelines actually, managing the performance and costs. But then we have our domain experts, subject matter experts, AI product managers. These are the ones that actually knew what the user experience would be. they probably are super familiar with the principles that we're actually building to our AI applications. They're tracking our evals and they're really trying to ensure that the product success. So there's this split between responsibilities but everybody is contributing but then there's this difference in terms of maybe technical abilities. And so with prompt learning it's going to be a combination of all these things. So everybody's going to really need to be involved and we can talk about that a little bit more. So what even is prompting? I'm going to first go through some of the approaches that we borrowed when we came up with prompt learning. So this is something that Arize has been really dedicated to doing some research. And so one of the first things we borrow from which is reinforcement learning. How many folks here are familiar with how reinforcement learning works? All right, cool. So if I were to give a really silly analogy, we have a reinforcement model. Pretend it's a student brain that we're trying to boost up. And so they're going to take an action which might be something like you're just going to take a test an exam and there's going to be a score. A teacher is going to come through and actually score the exam here that's going to produce this kind of scaler reward and pretend the student has an algorithm in their brain that can just take those scores and update the weights in their brain and the learning behavior there and then we reprocess. So in this reinforcement one we're updating weights based off of some scalers. But it's really actually difficult to update the weights directly, especially in the LLM world. So, reinforcement learning isn't going to quite work that well when we're doing things like prompting. So, then there's metaprompting, which is very close to what we do with prompt learning, but still not quite right. So, here with metal prompting, we're asking LM to improve the prompt. So again, we use that student example. We have an agent which is our student. And it's going to produce some output that's a user asking a question getting an output. That's our test in this example. And then we're going to score. Eval is pretty much what you can think of there. Where it's going to output a score and from there we have the metapromp thing. So now the teacher is the metapar prompt. It's going to take the result from our scorer and update the prompts based off of that. But it's still not quite what we want to do. And that's where we introduce this idea of prompt learning. So prompt learning is going to take the exam going to produce an output. We're going to have our LLM evals on there. But there's also this really important piece which is the English feedback. So which answers were wrong? Why were the answers wrong? Where the student needs to actually study? Really pinpointing those issues. And then we still aren't using metapro. We still are asking an LLM to improve the prompt. It's just the information that we are giving that LLM is quite different. And so we're going to update the prompt there with all of this feedback. So from our evals from a subject matter expert going in and labeling and use that to boost our prompt with better instructions and sometimes exams.

SallyAnn DeLucia: So this is the traditional prompt optimization where we're treating it like an ML where we have our data and we have the prompt. We're saying optimize this prompt and maximize our prediction impulse. But that doesn't quite work for LLMs, we're missing a lot of context. So what we really found is that the human instructions of why it failed. So imagine you have your application data, your traces, a data set, whatever it is. Your subject matter expert goes in and they're not only annotating correct or incorrect. They're saying this is why this is wrong. It failed to adhere to this key instruction. It didn't adhere to the context. It's missing out whatever it is. And then you also have your LLM explanations from LLM as a judge, which is same principle where instead of just the label, it provides the reasoning behind the label. And then we're pointing it at the exact instructions to change. We're changing the system prompt to help it improve so that we then get prediction labels, but we also get those evals and explanations of it. So, we're just optimizing more than just our outlet here. And I think a really key learning that we've had is the explanations in human instructions or through your own as a judge. That text is really valuable. I think that's what we see not being utilized in a lot of other broad optimization approaches. They're either optimizing for a score or they're just paying attention to the output. But you can think of it this way. It's these elements are operating in the text domain. So we have all this rich text that tells us exactly what it needs to do to improve. why wouldn't we use that to actually improve our so that's the basics of prompt learning but everybody always comes up to me and sounds great but does it actually work it does and we have some examples of when we do this so we did a little bit of a case study. I think coding agents everybody is pretty much using them at this point there's quite a few that have been really successful. I think Claude Code is a great example Cursor but there's also CL which is more of an open version of this and so we decided to take a look and compare to see if we could do anything to improve. So these are the baseline of where we started here. You can see the difference between the different models. Obviously using GPT-4 and Claude kind of the state-of-the-art there but we also had this opportunity where CL was using GPT-4o and it was working decently well at 30% versus 40. And then there was the conversation around. So this is where we started and we took a pass optimizing the system prompt here. So you can see this is what the old one was looking like. It has no rules section. So it was just very you are a cloud agent. You're built on this model. You're here to do coding. But there was no rules and so we took a pass at updating the system. So there were all of these different rules associated. So when dealing with errors or exceptions, handle them in a specific way. make sure that the changes align with the systems design. Any changes to be accompanied by appropriate test. So really just building in the rules that a good engineer would have which was completely missing before. And so we found that plan performs better with updated system problem. Pretty simple. It's the whole concept here. It's you can see these different problems and we're seeing things that were incorrect now being correctly done just by simply adding more instructions. So it really demonstrates pretty well here how those system prompts can improve and we benchmarked again with SWE-bench Lite to get another coding benchmark for these coding agents and we were able to improve by 15% just through the addition of rules. So I think that's pretty powerful. So no fine-tuning, no tool changes, no architecture changes. I think those are the big things folks reach for when they're trying to improve their agents. But sometimes it's just about your system prompt and just adding rules. I think we've really seen that and that's why we're really passionate about prompt learning and prompt optimization in general is it feels like the lowest lift way to get massive improvement gains in your agent. GPT-4 achieved performance near GPT-4o which is pretty much considered right now state-of-the-art when it comes to coding questions and it's twothirds of the cost which is always really beneficial. So these are some of the tables here. will definitely distribute this so you can take a closer look. But I think the main point I want y'all to come away with is the fact that 15% is pretty powerful improvement in our performance.

SallyAnn DeLucia: Now, a question we get all the time is we're taking these examples of perform learning. So, how this is really important is we're going to take a data set. A lot of time that data set is going to be a set of examples that didn't perform well. either a human went through and labeled them and found that they were incorrect or you have your evals that are labeling them incorrect and so you've gathered all these examples and that's what we're going to use to optimize our prompt. So I get a question all the time well aren't we going to overfit based off of these bad examples but there's this rule of generalization where mending properly enforces high level reusable coding standards rather than repo specific fixes and we are doing this train test split to ensure that the rules are generalized beyond just local quirks and whatever our training data set is. But if you think of this as you hire an engineer, right, to be an engineer at your company, you do want them to overfit to the database that they're working on. So, we feel that overfitting is maybe a better term for it is expertise. We are again not training in the traditional world. We are trying to build expertise and as we'll talk about this is not something we feel that you do once. You're actually going to continuously be running this. So, more problems are going to come up. we're going to optimize our prompt for what the application is seeing now. And then we'll So, we don't actually think it's a flaw. We feel like it's expertise instead. We can adapt as needed and mirroring what humans would do if they were taking on a task themselves. This is just another set of benchmarking again proving here that this diverse evaluation suite that focuses on the task for those difficult or tasks that are difficult for large language models and we're seeing again success with our improvements.

SallyAnn DeLucia: Now GABA just came out recently and I think that's something everybody's really excited about. I think the previous DSP optimizers were a little bit more focused on optimizing a metric and as we talked about we really want to be using the text modality that these applications are working in that have a lot of the reasons or how we need to improve and so we definitely wanted to do some benchmarking here. So how many people are familiar with GABA about it? All right, cool. Well, I'll just give a high level. I just noted that the main difference between their other new pro optimizers is that they are actually using this positive reflection and evaluation while they are doing the optimization. So it's this evolutionary optimization where there's this Pareto-based candidate selection and probabilistic merging of prompts. What this really does under the hood is we take candidate cross we evaluate them. Then there's this reflection LLM that's reviewing the evaluations and then making some mutations some changes and repeating until it feels like it has the right set of prompts. So I think something that is important to notice about GABA is it doesn't really choose just one. It does try to keep the top candidates and then do the merging from there. But we benchmarked it and proper learning actually does do a better job. And I think something that's really key is it does it in a lower number of loops. And I think something that we'll talk about in just a second here is that it does actually matter what your evals look like and how reliable those are. I think that's something we really feel strongly about at Arise is you definitely want to be optimizing your agent prompts, but I think a lot of people forget about the fact that you should also be optimizing your eval prompts because if you're using evals as a signal, you can't really rely on them if you don't feel confident in them. So, it's just as important to invest there, making sure you're applying the same principles that you are to your agent prompt as your eval prompts so you have a really reliable signal that you can trust and then feed that into your prompt optimization. But, in both of these graphs, the pink line is prompt learning. We did also benchmark it against MetaPrompt their older optimization technique that I was mentioning functions off optimizing around score and eval make the difference. So I highlighted on this slide here that with eval engineering we were able to do this. So we did have to make sure that the eval part of prompt learning were really high quality because again it's this only works if the eval itself is working. So, yep, evals make all the difference. Spend some time optimizing a prompt here. Again, it's all about making sure you have proper instruction. The same rules apply.

SallyAnn DeLucia: So, I want to walk through. I know there's a lot of content. I think it's really important to have context. But before we jump into any of the workshops, any questions I could answer about what I discussed so far?

Audience 1: Uh, I have a question comment. So I I think coding is the greatest example in terms of having the structure and evals. One thing I'm curious about is if you have other examples general prompts for general interactions with systems that are not as easily quantifiable. I'm just curious about any experience you guys have there.

SallyAnn DeLucia: Yeah. Is that for eval general?

Audience 1: Well I think it's just clear how you would set up what the eval would look like and I'm just wondering how you would do that for other types of so the question is is there any instruction for how you should set up your evals? coding seems a very straightforward example. You want to make sure the code's correct, right? But where some of these other agent tasks it's a little bit harder. I think the advice that I usually give folks is we do have a set of out of box. You can always start with things like QA correctness or focus on the task. But what I always suggest is getting all the stakeholders in the room. So getting those subject matter experts and security leadership and really defining what success would look like and then start converting that to different evaluations. So I think an example is Sterling and Alex. I have some task level evaluation. So I really care did it find the right data that it should have. Did it create a filter using semantic search or structured making the right tool call? And then I care did it call things in the right order? Was the plan correct? So thinking about what each step was and then even security will be well we care how often people are trying to jailbreak Alex. So, it's just taking each of those success criteria, converting it to eval. And we do have different tools that can help you, but that's usually the framework I give folks is start with just success and then worry about converting into an eval after.

Fuad Ali: Yeah. Just to add to that, maybe more of a subjective use case is for example Booking.com is one of our clients and so when they do what is a good posting for a property what is a good picture? Defining that is really hard, right? To you, you might think something is a very attractive posting for a hotel or something, right? But to someone else, it might look really different. And sometimes, as Sally was alluding to, it's sufficient to just gate it as a good, bad, and then iterate from there. So is this a good picture or bad picture? Let decide and then gate from there into specific background, this was dimly lit, the layout of the room was different, etc., etc. Yeah.

Audience 2: Yeah. That's that you're actually building on the question I was going to ask which is that they end up with that binary outcome which doesn't necessarily give you a gradient to advance upon are you then effectively using those questions dimly lit not to get a more continuous space is that

Fuad Ali: exactly right and then from there as you get more signal you can refine your evaluator further and further and then use those states and you can actually put a lot of that in your prompting itself right so yeah

Audience 3: I have two questions and I'm not sure if I should ask both of them or maybe your workshop will answer it. One is about rules and the rule section or operating procedures. I'm curious how you do you just continuously refine that in the English language and maybe reduce the friction of any contradictory rules. That's the first question. And then the other was I would love to see the slide on eval. if you could just say a little bit more on how you approach that because my issue in doing this work is whether or not to have an a simulator of the product and then the simulator is evaluating or to do what I'd like to do which is an end toend evaluation that I build but I would love to see you talk about that if you could.

SallyAnn DeLucia: Yeah, absolutely. So from the first one about how the instructions it's definitely something I think that you iterate over time on them. So a lot of times I think we take our best bet we write them by hand, right? And I think what we're trying to do with proper optimization is leverage the data to dynamically change them. And is I think great at removing redundant instructions, things like that. But the goal is we want to move away from static instructions. We feel very confidently that that is not going to really scale. It's not going to lead to sustainable performance. So the idea exactly with prompt learning is something that you can run over time. We see this even a long running task eventually where you're building up examples of incorrect things maybe having a human annotate them and then the task is always running producing optimized prompts that you can then pull in production and it it is a cycle that repeats over time.

Audience 3: Sorry just to intervene. So, are you saying that when you're doing this over a long period of time and then you have examples, you're just running the shots back into your rules section?

SallyAnn DeLucia: Kind of. It's going to pass it when we get to the optimization actual loop we're going to build, you'll see it as you are feeding the data in that's going to build a new set of instructions that you would then, push to production to use.

Audience 3: Okay.

SallyAnn DeLucia: I think your second question was around evals and how to where to start, how to write them and how to optimize those. Is that right?

Audience 3: Yes.

SallyAnn DeLucia: Yeah. So, it's a very similar approach. I think it's the data that you're reviewing is almost a little bit different. So, I should have pulled up the the loops. I don't know if you can find it. Let me just try something really quick to show this. There we go. So, this is how we see it is you have two co-evolving loops. I've been talking about the one on the left, the blue one a lot about we're improving agent, we're collecting failures, setting that to do fine-tuning or prompt learning, but you basically want to do the same thing with your evals where we're collecting the data set of failures, but instead of thinking about the failures being the output of your agent, we're actually talking about the eval output. So having somebody go through and evaluate the evaluators or using things like log probabilities as confidence scores or LLM as a judge to determine where things are not confident. We're doing the same thing. So figuring out where your eval is low confidence and then you're collecting that annotating maybe having somebody go through and say okay this is where the eval went wrong. And so it's the same pretty much process of optimizing your eval prompt. It's just I think folks think they can just grab something off the shelf or write something once and then they can just forget about it. But this loop, I've said it a few times, but the the left loop only works as well as your eval.

Audience 3: Sorry, I think my question is actually way more static and basic. It's do you are you talking about this orange circle as are you building a system or simulator for the eval or are you just talking about system prompt, user prompt, eval?

SallyAnn DeLucia: Yeah, I think it's more right now what we're talking about is just the different prompts. You could definitely do simulation, but I think that's a whole different workshop.

Audience 3: Thank you. Any more maybe questions before we get to the bridge club? Any switch back?

SallyAnn DeLucia: All right. So here is going to be a QR code for our prompt learning repo. So I'll give everyone a few minutes to get such with that. Get it on your laptops. I know it's a little bit clunky to add this QR and AirDrop it. was not sure a better way. I can just show you also here if you want to find it. It is going to be in our Rise AI repo here and under prompt learning and you just want to clone that. We are going to be running it locally here.

Audience 4: You go back to the page with the URL.

SallyAnn DeLucia: Yes. Sorry about that.

Audience 4: Oh no, the page with the URL. Oh,

SallyAnn DeLucia: we'll give folks just a few minutes to get

Audience 5: What do you What's your process when you're building a new agent or work for anything that could be evaluated? Do you guys start by just try something prototype and then see where it's bad and then do eval?

SallyAnn DeLucia: Yeah, I think there's different perspectives on this. Our perspective is Evals should never block you. You need to get started and you need to just build something really scrappy. We don't think you should waste time doing eval. I think it's helpful to pull something out of the box sometimes in those situations just because it's hard to comb through your data. that's something we've experienced with Alex when you're getting started just running a test manually reviewing it's painful. So I think that having eval is helpful but shouldn't be a blocker. Pull something off the shelf maybe start with that then as you're iterating you're understanding where your issues are then you're starting to refine your evals as you're refining your agent.

Audience 5: Yeah. One last question. Yeah.

Audience 6: So it makes sense to optimize the system sub-agents or commands or how are you thinking about this multi-agent?

SallyAnn DeLucia: Yeah. So the question is are you just doing one single prompt or how do you think about this in a multi-agent? I think we're thinking that this right now is independent tasks that can optimize your prompts independently and then running tests to get into the agent simulation of running them all together. But right now, our approach is a little bit isolated, but I definitely see a future where we're going to meet the standard of sub-agents and everything else that's going on right now.

Fuad Ali: No, I think that's pretty accurate. And also I mean even in a single agent use case versus a multi-agent use case ultimately each of those agents may be specialized. They may have their own prompts that they need to learn from. So I think doing this in isolation still has benefits for the multi-agent system as a whole that can pass on over time in scenarios hand off etc and making something really specialized. So I guess what we're talking about with the overfitting as well which is again question we get all the time but really you want to be overfit on your code base as an engineer. You don't want to be so generalized that you're no longer good at picking up specific works in your code base. Yeah.

SallyAnn DeLucia: All right. Everybody getting to read the mode. Okay. Anybody need any help?

SallyAnn DeLucia: All right. So, we are going to be using OpenAI for this. So, I think the next thing that I'll have everyone do is probably spend some time just grabbing your API key. We'll get to it and then I'll start walking through our notebook here. So, we are going to be doing a JSON webpage prompt example. So, you're going to find that under notebooks here. And so we'll give everybody a second to pull it out. There's going to be just some slight adjustments we're going to add to this example just to make it run a little faster and work a little better. The first is what this is even doing this is going to be a very simple example for just a JSON web page prompts. If anybody has a prompt or use case that they want to code along, Fuad and I are absolutely glad to help adapt what you're working on to the use case here. It's something very simple just to demonstrate the principles and we are going to be using we can definitely experiment. If you want to swap out any other providers that you want to use, we can also definitely help you do that. But the goal of this is essentially going to be to iterate through different versions of a prompt using a data set. And we will optimize. So the first thing is obviously we need to do some installs. I am just going to have you all update it. It says greater than 2.0.0. But we're going to actually just use I think 2.2 today. And then the next thing is just to make this run a little faster. So we're going to run things in async which is missing. So you can go ahead and add these lines in the cell as well. All right, everyone follow along and I never want to move too fast. Seems to head nods. Cool. Let's talk about configuration. So I talked about it a little bit when I was going through the slides. So we are going to be doing some looping. So the general idea is we start out with the data set with some feedback in it and we'll look through the data set once we get it. But you're going to want to have either human evaluation. So annotations, either free text, labels, or you're going to want to have some evaluation data. But the feedback is really important. That's what makes this work. We're going to then, pass that to LLM to do the optimization and then it's going to basically have eval. So as it's optimizing, it's using that data set to then run and assess whether or not it should keep optimizing. And then it also provides you data that you can use to gauge which of the prompts that it outputs in a production setting. So we're going to do some configuration. So I've wrote out here what each of these means. So we have the number of samples. So this controls how many rows of the sample data set. You can set to zero to use all data or you can use a positive number to limit for faster experimentation. So I think that sometimes folks use different approaches here. Sometimes you want to just move really quick so you set a low sample. Sometimes you want to be a little bit more representative so you up it. I have it here set as 100. Feel free to adjust. And then the next thing is train split. So I think folks are probably pretty familiar with the concept here of a train test split, but it's just how much of the data do we want to use into our training? Again, that's what we're using to actually optimize. Then how much of it do we want to use when we're testing when we're running the eval on the new prompt? And there's number of rules. Basically the specific number of rules to use for evaluation. This just determines which prompts to use. And so this is as we're running these loops, we're outputting a bunch of different prompts. So this is just saying how many we should use for evaluation. And then key one here, number of optimization loops. So this sets how many optimization iterations to run per experiment. And each loop basically generates those outputs, evaluates them and refineses the prompt. And so these just control the experiment scope the data splitting just went through the whole prompt learning loop and and how much data we want to use. So you can just run these as you are or if you want to adjust them feel free. And then the next step pretty simple. We're just going to grab that open AI key if you haven't already set that up. So, get passage is going to pop up. I'll show you here quick. It's going to pop up there. You can just paste in your API key there before we start looking at the the data a little bit. Just if anybody runs into any issues, you just give this away. All right. I think this particular we get through this

Audience 7: I'm doing good but if you have a free one you want to give me that

SallyAnn DeLucia: I wish

Audience 7: all right let's talk about the data so we provided data with you with queries you can see here that we're doing the 8020 split based off of configuration we set above I'm just going to pull this train set here and let's just

Audience 8: Yeah, I run because in the minus 50

SallyAnn DeLucia: Oh, yep. You're right. That's a mistake on my part. Yeah, it is the 50. Let's take a look at what this data set looks like. No. Just so folks can understand. So starting here with some just basic input and output. Transcript we don't have any of the feedback in these rows that I printed out here but you can imagine you can have different correctness labels here explanations any real validation data can be whatever it is that you'd like it to be. Some folks use multiple eval feedback sometimes it's a combination but you really want to have the input and output that will use that way. Should my output of train set be the same as you?

Audience 9: Not necessarily. Depends on

SallyAnn DeLucia: I didn't know if head was sort or not.

Audience 9: It all depends on what the the same but we could look at if I did this this should be the same for you maybe just to make sure.

Audience 10: Yeah. That's what you're saying. Okay. Yeah. Quick question. Um, is it possible for the input to be a chat history and not just

SallyAnn DeLucia: Great question. So, I think it depends on what it is you're trying to do. If you're doing just a simple system of the input, you want it to be one to one. You don't want to give it a ton of conversation data that's not relevant to the prompt that you're optimizing. we generally just use the single input but I think that there are applications that you could do conversation level inputs.

Audience 10: Yeah. Because because quite often the failure is somewhere mid-conversation right. So so if you put just the original task in then the probability of you hitting a failure in the middle of the

SallyAnn DeLucia: totally. So in that case, what we generally see is different rows of having each of the back and forths be independent rows because you're probably going to evaluate each of them and honestly probably get the human feedback on each of them. So we usually separate them out in that way.

Audience 10: But it's a good point. If you just always are focusing on the first turn, there's probably a lot of redundancy there. you definitely will have to say over parts of the conversation

Audience 11: and and how we can differentiate instructions and we have some context also. So

Audience 12: should not touch the context. It should only whatever the manipulate the system instruction or the prompting context it should be the static it should not be based on the answer it will change my context.

SallyAnn DeLucia: Yes. What you're saying is looking at the input there might be a tool volume context you're passing that in. You can absolutely include that in your data set so that the application understands what other or not the application but the prompt learning LLM can understand all of the data that's available. So you can just have that passed in as extra column if you want. Most people start with just input and the feedback. But you can absolutely add what other data you think is relevant and if when for the rerunning when we're doing the experiment of testing you'll definitely always want to have the data that would be required to answer

Audience 12: any even very simple some call some call or some context it is pulling some API call whatever the prompt engineering it should be based on the out getting the output right and whatever the context front my plus whatever the tool call I have done API call all the context engineering and then last finalize

SallyAnn DeLucia: totally yeah so again at this point we're testing just one prompt and not that end to end but you definitely want to have everything that is flowing into the prompt that you're optimizing so if your system prompt takes in the user input for example some data from an external API you would definitely want to provide all of that data does that make sense Because you're saying that the the trajectories the tool calls and what the agent's going to do depending on what the tool call was is what you're trying to optimize.

Audience 12: Yeah, exactly. We want to just because we're trying to replay and optimize one step of it. We definitely don't want to do it completely in isolation. So if there's data that flows into that prompt that's context that's using that's producing the output, right? So we want to be sure that we're including that. We don't want to exclude anything. But if it's data that comes at a different step probably not then you don't want to do that that way. It's just think about what's relevant for the the step that we're trying to optimize in this.

SallyAnn DeLucia: All right. Any other questions coming? All right. Cool. So we're going to set up our initial system prompt. You can see this is something very basic. We'll definitely I think we can do a whole lot better than this, but I just want to illustrate something that we're going to test and optimize. So we're just saying you are an expert in JSON web page creation. Your task is input. And then so all these inputs that we're seeing are going to be what we're actually generating outputs for and trying to optimize. Now I already touched on this. Evaluators are extremely important to make all of this work, right? So we're going to initialize two evaluators that use LLM as a judge to assess the quality of generated outputs. So we are using LLM as a judge. If you have any other codebased evaluations, whatever you need to do to evaluate, you can definitely swap those out. But we're going to do evaluate output. This is going to be a comprehensive evaluator that assesses the JSON webpage correctness against the input query and the evaluation rules. It's going to provide an output label of correct or incorrect. So pretty simple binary. Again, you can use multi-label. And then it's going to have the detailed explanations as well. And then we have a rule checker. This is a more specialized evaluator that performs a granular rule by rule analysis. And it examines if each rule was compliant. And then both of these are going to generate feedback that goes into our optimization loop to iteratively improve the system prompt. Explanation, rule violations guide. And we'll get to this the prompt learning optimizer and creating the more effective prompts. So I have some imports here. Let's take a look at what the actual eval output has. So we do have some rules that are in in here wait they're going to be in a repo. So we're going to open that as a file. We have this llm provider and we're using open AI here. And then we're going to do our classification evaluator. So, we're just calling it evaluate output. It's al we have an evaluation template that we're reading from the bottom here. Then we just have choices correct and correct. Now we're mapping a label to a score. Sometimes it's helpful to be able to add or score. Sometimes a number is easier than just looking at a bunch of labels. It is optional you want to map these if you have a multiclass use case. You can set the scores accordingly. But these are just going to be our choices the rails that we want our LLM as a judge to adhere to. And then all we're doing here is getting our results. I have it doing some printing so you can take a look. So this is going to be slightly different than what you're seeing in the notebook. So I'm just going to pause here. If you want to make the code changes from what you're seeing in probably your version, this is a good time for that. Does the setup of the evaluator make sense to all the key. It's going to be the rails. It's going to be the output. And of course our template.

Audience 13: Yeah, you will want to grab your own OpenAI key here to set

SallyAnn DeLucia: and we can help you if you want to use different provider. We can help you swap this out that is helpful to anybody.

SallyAnn DeLucia: Okay, I'm going to start walking you through the output generation. So this is just you can imagine this as your own agent logic or the the part that you're testing. This is just going to function that actually generates the JSON outputs. We're using for one here with JSON response format zero temperature for consistent outputs. It's taking a data set a system prompt generates outputs for all rows returns the results for evaluation. And it's called during each iteration to produce output. So this is our experimentation function that we're writing. So as we're passing in data, it's producing new prompts. We need a way to test it, evaluate, understand how we are moving the needle here. So that's all this is. So it's pretty straightforward function just called generate output. We have that output model. Again, we're using OpenAI. If anybody wants help switching things around, happy to help. We are using response format because we are dealing with JSON here. So we know that what you just prompted. I mean some of the the newer models are decent at it, but using response format is really helpful. And then we're also setting temperature to zero. And here is where we're passing all the data in. So the data set because again we want to run this on all of the the testing data, the system prompt that will be input. So as we get to the optimization loop, we're going to be passing in a new prompt to this with the data set and then evaluating. We have our output model that we've already passed, concurrency, all that good stuff. And it's just returning all of the outputs there. Would you for the current generation of models since this one's basically in AI terms ancient would you still recommend setting the temperature to zero or would you actually want to try to encourage some of the creativity to

Audience 14: I think it depends on the use case a little bit and what you're you're trying to do. You can definitely experiment that and take it through the lens of how how important is consistency to use something I feel JSON web page I feel consistency probably temperature zero makes sense but I definitely think not for every agent every use case do you want to use zero

SallyAnn DeLucia: any other questions get moving all right additional metric so we talked about before that we are using some score mapping this part is optional you want to use the metrics that make sense to you we're not directly using this as we are using it to know whether or not we optimize but it's not we're using this as our sole indicator for the success. Here we are just going to calculate some very basic metrics. It's just you can choose something accuracy, F1 precision, recall just some basic classification metrics for us to understand and because we are using binary mapping scores we can do that. And so that's what you're seeing happen here. We're mapping to binary and then just based off the score we calculate the metric. So very simple helper function here.

SallyAnn DeLucia: All right, the good stuff the optimization loop. We made it. Okay, so this cell implements the core prompt optimization algorithm. It's a three-part process. So we want to generate and evaluate. So generate outputs using the current prompt on the test data set and evaluate their correctness. We want to train and optimize. If results are unsatisfactory, generate outputs on the training set, evaluate them, use the feedback to improve the prompt, and then iterate. So we want to repeat until either the threshold is met or all the loops are completed. So if you remember above, we're setting that to just five loops. And then we can repeat based off of that or the thresh met it's going to track metrics across all the iterations. So turn to detailed results including a train test accuracy scores the optimized prompts and the raw value. So as I mentioned at the beginning as we're running these different loops on the experiments we're going to be producing a lot of different prompts. And so we're getting that information back that you can use. And these are our key parameters. I'll go through them, as we get to the code, but just to give you a heads up. This is the target accuracy score to stop optimizations. It could also be whatever other metric you'll see, we have a score so you can determine the number of loops of the optimization iterations. We've set that score and then the number of rules. Again, these are some configurations we've already set. Cool. So, optimization loop. This is going to take in all of those parameters that I've mentioned there. It just kicks off saying hey we're starting it's going to do the initial evaluation so we understand how things are starting off. Again you can pass in data too. You can skip this initial evaluation. We're running it at the start here. But if you were running production setting, you might already have evalu. And then it's going to assess the threshold against our initial valuation. Again, this could be skipped when we're coming from a production setting, but wanted to start us off from scratch so that we can get a real feel for this. And then it starts the loop. So, we're generating output. It's setting that as the train output. So, when I printed train, you saw the outputs. I skipped ahead there. And then it also will set correctness, explanation, any rule violations. And then we'll actually use our prompt learning optimizer. So this comes with the SDK the prompt learning SDK that you can use with the rise. So we're sending in that prompt optimization the best choice and then that API. So under the hood as we talked about in the slides taking in that feedback taking in the original prompt and trying to optimize to get better results and then spinning out prompt and then can also add an evaluator. So again those three feedback columns we're looking to get back as correctness explanation for that if there are any rule violations and then from there we just kicked off the optimizer and optimize with our train set output those feedback columns again and then any context size limitations you want to add next step so the optimizer again is going to take our data produce a prompt we want to evaluate so we understand how we're doing what this code block doing is doing here so trying to get that new prompting again with all those details getting our result and then we do that with our test set as well and then we're getting back our score and our metric value and then doing the checks and then we repeat it all again till we either get above our threshold or we've hit the max number of loops and then returning our results. So that's what's going to be happening other here. Any questions on that? just some result saving function more helper functions here. So we do want to obviously save all these results. We don't want them just be ephemeral that we can't ever access again. So just saving them all. You can also save all the single experimentation so you have all of that data towards the end. We'll be able to pull this and determine what the best prompt is. But these are just very basic helper functions. I don't spend too much time just saving them to CSV at the end of the day. Now we execution it. So this cell runs the prompt optimization experiment, saves the results. We're getting the JSON format, the CSV format. It includes calls for the iteration number, the number of rules, test, train, accuracy scores, all the data that we're actually going to need to evaluate whether or not this thing is successful, and then we're going to start getting results here. So, this does take quite a while to run. So, we'll run and I think this will be a great point for discussion, but as you are running it, you're going to start seeing the different loops. outputs coming out as well. And yeah, we'll just work through it as it it runs. It's probably going to take 20 30 minutes for things to run, but happy to take any questions and help anybody out as they run into issues.

Audience 15: One thing, can you scroll back to the part of code that we needed to change?

SallyAnn DeLucia: Change. It's gonna be

Audience 15: So, one reminder are running into this. I don't think I was

SallyAnn DeLucia: for this line here, when you're doing install, you do want to be equals 2.2. Because I think there's a a little bit of a package issue. So just make sure that's you're hitting errors with the eval. If not, let me down and try to fix it. This is the reason why

Audience 15: uses a generic evaluation.

SallyAnn DeLucia: Yes. And you can see the evaluation problem if you go to the We've just taken that part out of this, but we can definitely go through that. So if you look here on this line here, we're reading in under prompts here, you can find the evaluation if you're curious. And this is the reason why everyone hates on docker. This is why we use all.

Audience 16: Yes, absolutely.

Audience 17: The notebook.

SallyAnn DeLucia: So I would also recommend patching your code with nestio if you haven't already. Helps it run a lot faster. Also for the purpose of the workshop I switched our loops to one. that took me six minutes to run. So would recommend also doing that instead of having five. Obviously wouldn't recommend doing that when you're actually optimizing your prompt, but for now it'll help you get through the workshop.

SallyAnn DeLucia: All right, I just want to call out the the last little bit here.

Audience 18: the last step before folks, let's see. Okay. So the the last little bit of code here is just to extract the prompt that achieves the best test accuracy. So I mentioned how we're saving up all the results to use. We just have a function that essentially gets the last or the best version of that showing you the original and then the best optimized version which you can then use to pull and put into your code. I did want to just give one call out as you saw today can be a little bit difficult to to manage and so I want to call out for those of you who are looking for more of an enterprise solution to this in Arise you do have these prompt optimization tasks. You can have your prompts living in our prompt hub data sets with all of your human annotations or eval that you can either create from traces or just by ingesting it into Arise. And then from there, all you really need to do is give it a task name, choose what you want your training data set to be, where the output lives, where all your feedback columns are. You can adjust all of the parameters that you'd like. And then from there, you can just kick it off and it will produce an optimized prompt in the hub for you. So if I go over here, I think I have some. No, maybe not. it will basically just create a new version here that says it's optimized prompts with all the results and we are building on this so you can add all your evals to it have that all running in the loop but just wanted to call out that if you're not interested in maybe maintaining code loops and having to build a task infrastructure yourself it is something that we do offer in Arise but yeah hopefully I know some folks are hanging out we'll be sticking around here for a little while as we can help you work through issues But thanks so much for joining us. Hopefully you learned something useful.