Thanks everybody for uh for joining. I'm here to talk to you today about DSPI. Um, and feel free to jump in with questions or anything throughout the talk. It's, you know, I don't plan on spending the full hour and a half or so. I know it's the last session of the day. So, um, keep it casual. Feel free to jump in. I'll start with a little bit of background. Don't want to go through too many slides. I'm technically a consultant, so I have to do some slides, but we will dive into the code for the the latter half. And there's a GitHub repo that you can download to to follow along and play around with it on your own. Um, so how many people here have heard of DSPI? Almost everyone. That's awesome. How many people have actually used it kind of day-to-day in production or anything like that? Three. Okay, good. So hopefully we can convert some more of you today. Um, so high level DSPI, this is straight from the website. Um, it's a declarative framework for how you can build modular software. And most important for someone like myself, I'm not necessarily an engineer that is writing code all day, every day. As I mentioned before, I'm a more of a technical consultant. So, I run across a variety of different problems. Could be um an investigation for a law firm. It could be helping a company understand how to improve their processes, how to deploy AI internally. Maybe we need to look through 10 10,000 contracts to identify a particular clause um or or paragraph. And so DSPI has been a really nice way for me personally and my team to iterate really really quickly on building these applications. Most importantly building programs. It's not um it's not kind of iterating with prompts and tweaking things back and forth. It is building a a proper Python program and and DSP is a really good way for you to do that. So I mentioned before there's a repo online if you want to download it now and kind of just get everything set up. I'll put this on the screen later on. Um, but if you want to go here, just kind of download some of the code. It uh it's been put together over the past couple days. So, it's not going to be perfect production level code. It's much more of utilities and little things here and there to just come and kind of demonstrate the usefulness, demonstrate the point of of what we're talking about today in that and we'll walk through all of these these different use cases. So um sentiment classifier going through a PDF some multimodal work uh a very very simple web research agent detecting boundaries of a PDF document you'll see how to summarize basically arbitrary length text and then go into an optimizer uh with Jeepo but before we do that just again kind of level set the biggest thing for me personally DSP is a really nice way to decompose your logic into a program that treats LLMs as a first class citizen. So at the end of the day, you're fundamentally just calling a function that under the hood just happens to be an LLM and DSPI gives you a really nice intuitive easy way to do that with some guarantees about the input and output types. So of course there are structured outputs, of course there are other ways to do this, Pyantic [snorts] and others. Um, but DSPI has a set of primitives that when you put it all together allows you to build a cohesive modular piece of software that you then happen to be able to optimize. We'll get into that uh in a minute.

So, just a few reasons of why I'm such an advocate. It sit at it sits at this really nice level of abstraction. So, it's I I would say it doesn't get in your way as much as a lang chain. And that's not a knock-on lang chain. It's just a different kind of paradigm in the way that DSPI is is structured. Um, and allows you to focus on things that actually matter. So you're not writing choices zero messages content. You're not you're not doing string parser. You're not doing a bunch of stuff under the hood. You're just declaring your intent of how you want the program to operate, what you want your inputs and outputs to be. Because of this, it allows you to create computer programs. As I mentioned before, not just tweaking strings and sending them back and forth. You are building a program first. It just happens to also use LLMs. And really the the most kind of important part of this is that and Omar the KB the uh the founder of this or the the original developer of it had this really good podcast with A16Z. I think it came out just like two or three days ago. But it he put it a really nice way. He said it's a it's built with a systems mindset and it's really about how you're encoding or expressing your intent of what you want to do most importantly in a way that's transferable. So the the design of your system, I would imagine, or your program isn't going to move necessarily as quickly as maybe the model capabilities are under the hood. when we see new releases almost every single day, different capabilities, better models and so DSPI allows you to structure it in a way that retains the control flow uh retains the intent of your system, your program um while allowing you to bounce from model to model to the extent that you want to or need to. Convenience comes for free. There's no parsing, JSON, things like that. It again, it sits at a nice level of abstraction where you can still understand what's going on under the hood. If you want to, you can go in and tweak things, but it allows you to to kind of focus on just what you want to do while retaining the level of precision that you that I think most of us would like to have in and kind of building your programs. Um, [snorts] as mentioned, it's robust to kind of model and paradigm shifts. So, you can again keep the logic of your program. Um but it but keep that those LLMs infused in uh basically in line. Now that being said, you know, there are absolutely other great libraries out there. Pedantic AI, Langchain, there's many many others that allow you to do similar things. Agno is another one. Um this is just one perspective and um it may not be perfect for your use case. For me, it took me a little bit to kind of gro how DSPI works and you'll see why that is in a minute. Um, so I would just recommend kind of have an have an open mind, play with it. Um, run the code, tweak the code, do whatever you need to do. Um, and just see how it might work, might work for you. And really, this talk is more about ways that I found it useful. It's not a dissertation on the ins and outs of every nook and cranny of DSPI. It's more of, you know, I've run into these problems myself now. I naturally run to DSPI to solve them. And this is kind of why. And the hope is that you can extrapolate some of this to your own use cases. So we we'll go through everything uh fairly quickly here, but the core concepts of DSPI really comes down to arguably five or these six that you see on the screen here. So we'll go into each of these in more detail, but high level signatures specify what you want the L what basically what you want your function call to do. This is when you specify your inputs, your outputs. Inputs and outputs can both be typed. Um, and you defer the rest of the basically the how the implementation of it to the LLM. And we'll see how we how that all kind of comes together uh in a minute. Modules themselves are ways to logically structure your program. They're based off of signatures. So, a module can have one or more signatures embedded within it in addition to uh additional logic. and it's based off of um pietorrch and some of the in terms of like the methodology for how it's structured and you'll you'll see how that uh comes to be in a minute. Tools we're all familiar with tools MCP and others and really tools fundamentally as DSPI looks at them are just Python functions. So it's just a way for you to very easily expose Python functions to the LLM within the DSP kind of ecosystem if you will. um

So signature as I mentioned before it's how you express your intent your declarative intent can be super simple strings and this is the weirdest part for me initially but is one of the most powerful parts uh of it now or it can be more complicated class-based classbased objects if you've used pyantic it that's basically what what it runs on under the hood. So this is an example of one of the class-based signatures. Again, it it's basically just a pyantic object. What's super interesting about this is that the the names of the fields themselves act almost as like mini prompts. It's part of the prompt itself. And you'll see how this comes to life in a minute. But what's ultimately passed to the model from something like this is it will say okay your inputs are going to be a parameter called text and it's based off of the name of the that particular parameter in this class. And so these things are actually passed through. And so it's it's very important uh to be able to name your parameters in a way that is intuitive for the model to be able to pick it up. Um, and you can add some additional context or what have you in the description field here. So most of this, if not all of this, yes, it is proper, you know, typed Python code, but it's also it also serves almost as a prompt ultimately that feeds into the model. Um, and that's basically translated through the use of adapters. Um, and so just to highlight here like these, it's the ones that are a little bit darker and bold, you know, those are the things that are effectively part of the prompt. uh that's been sent in and you'll see kind of how DSPI works with all this and formats it in a way that again allow you to just worry about what you want. Worry about constructing your signature instead of figuring out how best to word something in the prompt. Go >> ahead. I have a really good prompt. >> Sure. Then I don't want this thing. >> That's exactly right. >> Sure. >> So the the question for folks online is what if I already have a great prompt? I've done all this work. I'm a I'm a amazing prompt engineer. I don't want my job to go away or whatever. Um, yes. So, you can absolutely start with a custom prompt or something that you have demonstrated works really well. And you're exactly right that's that can be done in the dock string itself. There's there's some other methods in order uh for you to inject basically system instructions or add additional things at certain parts of the ultimate prompt and or of course you can just inject it in the in the final string anyway. I mean it's just you know a string that is constructed by VSPI. So um absolutely this doesn't necessarily prevent you it does does not prevent you from adding in some super prompt that you already have. Absolutely. Um and to your point it is it can serve as a nice starting point from which to build the rest of the system. Here's a shorthand version of the same exact thing which to me the first time I saw this so this was like baffling to me. Um, but it it that's exactly how it works is that you're basically again kind of deferring the implementation or the logic or what have you to DSPI and the model to basically figure out what you want to do. So in this case, if I want a super super simple text uh sentiment classifier, this is basically all you need. You're just saying, okay, I'm going to give you text as an input. I want the sentiment as an integer as the output. Now you probably want to specify some additional instructions to say okay your sentiment you know a lower number means negative you know a higher number is more positive sentiment etc. But it just gives you a nice kind of easy way to to kind of scaffold these things out in a way that you don't have to worry about like you know creating this whole prompt from hand. It's like okay I just want to see how this works and then if it works then I can add the additional instructions then I can create a module out of it or you know whatever it might be. It's these shorthand or it is this shorthand that makes experimentation and iteration incredibly quick. So modules it's that base abstraction layer for DSPI programs. There are a bunch of modules that are built in and these are a collection of kind of prompting techniques if you will and you can always create your own module. So to the question before, if you have something that you know works really well, sure yeah, put it in the module. That's now the kind of the base assumption, the base module that others can build off of. And all of DSPI is meant to be composable, optimizable, and when you deconstruct your business logic or whatever you're trying to achieve by using these different primitives, it all it's intended to kind of fit together and flow together. Um, and we'll get to optimizers in a minute, but at least for me and my team's experience, just being able to logically separate the different components of a program, but basically inlining uh LLM calls has been incredibly powerful for us. And it's just an added benefit that at the end of the day, because we're just kind of in the DSPI paradigm, we happen to also be able to optimize it at the end of the day. Uh, so it comes with a bunch of standard ones built in. I I don't use some of these bottom ones as much, although it's they're super interesting. Um the base one at the top there is just DSpi.predict. That's literally just, you know, an LM call. That's just uh a vanilla call. chain of thought uh probably isn't isn't as relevant anymore these days because models have kind of ironed those out but um it is a good example of the types of um kind of prompting techniques that can be built into some of these modules um and basically all this does is add um some some of the uh strings from literature to say okay let's think step by step or whatever that might be same thing for react and codeact react is basically the way that you expose the tools to the model. So, it's wrapping and doing some things under the hood with um basically taking your signatures and uh it's injecting the Python functions that you've given it as tools and basically React is how you do tool calling in DSP. Program with thought is uh is pretty cool. It kind of forces the model to think in code and then we'll return the result. Um, and you can give it a, it comes with a Python interpreter built in, but you can give it some custom one, some type of custom harness if you wanted to. Um, I haven't played with that one too too much, but it is super interesting. If you have like a highly technical problem or workflow or something like that where you want the model to inject reasoning in code at certain parts of your pipeline, that's that's kind of an really easy way to do it. And then some of these other ones are basically just different methodologies for comparing outputs or running things in parallel. So here's what one looks like. Again, it's it's fairly simple. It's, you know, it is a Python class at the end of the day. Um, and so you do some initial initialization up top. In this case, you're seeing the uh uh the shorthand signature up there. So, I'm this module uh just to give you some context is an excerpt from um one of the the Python um files that's in the repo is basically taking in a bunch of time entries and making sure that they adhere to certain standards, making sure that things are capitalized properly or that there are periods at the end of the sentences or whatever it might be. that's from a real client use case where they had hundreds of thousands of time entries and they needed to make sure that they all adhere to the same format. This was one way to to kind of do that very elegantly, at least in my opinion, was taking up top you can define the the signature. It's adding the some additional instructions that were defined elsewhere and then saying for this module the the change tense um call is going to be just a vanilla predict call. And then when you actually call the module, you enter into the forward function which you can inter basically intersperse the LLM call which would be the first one and then do some kind of hard-coded business logic beneath it. Uh tools as I mentioned before these are just vanilla kind of Python functions. It's the DSP's tool interface. So under the hood, DSPI uses light LLM. And so there needs to be some kind of coupling between the two, but fundamentally um any type of tool that would that you would use elsewhere, you can also use in in DSPI. And this is probably obvious to most of you, but here's just an example. You have two functions, get weather, search web. You include that with a signature. So in this case, I'm saying the signature is I'm going to give you a question. please give me an answer. I'm not even specifying the types. It's just going to infer what that means. Uh I'm giving it the get weather and the search web tools and I'm saying, okay, do your thing, but only go five rounds just so it doesn't spin off and do something crazy. And then a call here is literally just calling the React agent that I created above with the question, what's the weather like in Tokyo? We'll see an example of this in the code session, but basically what this would do is give the model the prompt, the tools, and let it do its thing.

um adapters live in between your signature and the LLM call itself. I mean, as we all know, prompts are ultimately just strings of text that are sent to the LLM. Signatures are a way for you to express your intent at a at a higher level. And so, adapters are the things that sit in between those two. So, it's how you translate your inputs and outputs into a format basically explodes out from your initial signature into a format that is ultimately the prompt that is sent to the LLM. And so, you know, there's some debate or some research on if certain models perform better with XML as an example or BAML or JSON or others. And so adapters give you a nice easy abstraction to to basically mix and match those at at will as you want. Optimizers um are the most interesting and for whatever reason the most controversial part of DSP. That's kind of the first thing that people think of or at least when they hear of DSP they think optimizers. We'll see a quote in a minute. It's not optimizers first. It is just a nice added benefit and a nice capability that DSPI offers in addition to the ability to structure your program with the signatures and modules and everything else. Um, and metrics are used in tandem with optimizers that that basically defines how you measure success in your in uh your DSPI program. So the optimizers use the metrics to determine if it's finding the right path if you will. So signature as I mentioned before it's how you express your intent your declarative intent can be super simple strings and this is the weirdest part for me initially but is one of the most powerful parts uh of it now or it can be more complicated class-based classbased objects if you've used pyantic it that's basically what what it runs on under the hood. So this is an example of one of the class-based signatures. Again, it it's basically just a pyantic object. What's super interesting about this is that the the names of the fields themselves act almost as like mini prompts. It's part of the prompt itself. And you'll see how this comes to life in a minute. But what's ultimately passed to the model from something like this is it will say okay your inputs are going to be a parameter called text and it's based off of the name of the that particular parameter in this class. And so these things are actually passed through. And so it's it's very important uh to be able to name your parameters in a way that is intuitive for the model to be able to pick it up. Um, and you can add some additional context or what have you in the description field here. So most of this, if not all of this, yes, it is proper, you know, typed Python code, but it's also it also serves almost as a prompt ultimately that feeds into the model. Um, and that's basically translated through the use of adapters. Um, and so just to highlight here like these, it's the ones that are a little bit darker and bold, you know, those are the things that are effectively part of the prompt. uh that's been sent in and you'll see kind of how DSPI works with all this and formats it in a way that again allow you to just worry about what you want. Worry about constructing your signature instead of figuring out how best to word something in the prompt. Go >> ahead. I have a really good prompt. >> Sure. Then I don't want this thing. >> That's exactly right. >> Sure. >> So the the question for folks online is what if I already have a great prompt? I've done all this work. I'm a I'm a amazing prompt engineer. I don't want my job to go away or whatever. Um, yes. So, you can absolutely start with a custom prompt or something that you have demonstrated works really well. And you're exactly right that's that can be done in the dock string itself. There's there's some other methods in order uh for you to inject basically system instructions or add additional things at certain parts of the ultimate prompt and or of course you can just inject it in the in the final string anyway. I mean it's just you know a string that is constructed by VSPI. So um absolutely this doesn't necessarily prevent you it does does not prevent you from adding in some super prompt that you already have. Absolutely. Um and to your point it is it can serve as a nice starting point from which to build the rest of the system. Here's a shorthand version of the same exact thing which to me the first time I saw this so this was like baffling to me. Um, but it it that's exactly how it works is that you're basically again kind of deferring the implementation or the logic or what have you to DSPI and the model to basically figure out what you want to do. So in this case, if I want a super super simple text uh sentiment classifier, this is basically all you need. You're just saying, okay, I'm going to give you text as an input. I want the sentiment as an integer as the output. Now you probably want to specify some additional instructions to say okay your sentiment you know a lower number means negative you know a higher number is more positive sentiment etc. But it just gives you a nice kind of easy way to to kind of scaffold these things out in a way that you don't have to worry about like you know creating this whole prompt from hand. It's like okay I just want to see how this works and then if it works then I can add the additional instructions then I can create a module out of it or you know whatever it might be. It's these shorthand or it is this shorthand that makes experimentation and iteration incredibly quick. So modules it's that base abstraction layer for DSPI programs. There are a bunch of modules that are built in and these are a collection of kind of prompting techniques if you will and you can always create your own module. So to the question before, if you have something that you know works really well, sure yeah, put it in the module. That's now the kind of the base assumption, the base module that others can build off of. And all of DSPI is meant to be composable, optimizable, and when you deconstruct your business logic or whatever you're trying to achieve by using these different primitives, it all it's intended to kind of fit together and flow together. Um, and we'll get to optimizers in a minute, but at least for me and my team's experience, just being able to logically separate the different components of a program, but basically inlining uh LLM calls has been incredibly powerful for us. And it's just an added benefit that at the end of the day, because we're just kind of in the DSPI paradigm, we happen to also be able to optimize it at the end of the day. Uh, so it comes with a bunch of standard ones built in. I I don't use some of these bottom ones as much, although it's they're super interesting. Um the base one at the top there is just DSpi.predict. That's literally just, you know, an LM call. That's just uh a vanilla call. chain of thought uh probably isn't isn't as relevant anymore these days because models have kind of ironed those out but um it is a good example of the types of um kind of prompting techniques that can be built into some of these modules um and basically all this does is add um some some of the uh strings from literature to say okay let's think step by step or whatever that might be same thing for react and codeact react is basically the way that you expose the tools to the model. So, it's wrapping and doing some things under the hood with um basically taking your signatures and uh it's injecting the Python functions that you've given it as tools and basically React is how you do tool calling in DSP. Program with thought is uh is pretty cool. It kind of forces the model to think in code and then we'll return the result. Um, and you can give it a, it comes with a Python interpreter built in, but you can give it some custom one, some type of custom harness if you wanted to. Um, I haven't played with that one too too much, but it is super interesting. If you have like a highly technical problem or workflow or something like that where you want the model to inject reasoning in code at certain parts of your pipeline, that's that's kind of an really easy way to do it. And then some of these other ones are basically just different methodologies for comparing outputs or running things in parallel. So here's what one looks like. Again, it's it's fairly simple. It's, you know, it is a Python class at the end of the day. Um, and so you do some initial initialization up top. In this case, you're seeing the uh uh the shorthand signature up there. So, I'm this module uh just to give you some context is an excerpt from um one of the the Python um files that's in the repo is basically taking in a bunch of time entries and making sure that they adhere to certain standards, making sure that things are capitalized properly or that there are periods at the end of the sentences or whatever it might be. that's from a real client use case where they had hundreds of thousands of time entries and they needed to make sure that they all adhere to the same format. This was one way to to kind of do that very elegantly, at least in my opinion, was taking up top you can define the the signature. It's adding the some additional instructions that were defined elsewhere and then saying for this module the the change tense um call is going to be just a vanilla predict call. And then when you actually call the module, you enter into the forward function which you can inter basically intersperse the LLM call which would be the first one and then do some kind of hard-coded business logic beneath it. Uh tools as I mentioned before these are just vanilla kind of Python functions. It's the DSP's tool interface. So under the hood, DSPI uses light LLM. And so there needs to be some kind of coupling between the two, but fundamentally um any type of tool that would that you would use elsewhere, you can also use in in DSPI. And this is probably obvious to most of you, but here's just an example. You have two functions, get weather, search web. You include that with a signature. So in this case, I'm saying the signature is I'm going to give you a question. please give me an answer. I'm not even specifying the types. It's just going to infer what that means. Uh I'm giving it the get weather and the search web tools and I'm saying, okay, do your thing, but only go five rounds just so it doesn't spin off and do something crazy. And then a call here is literally just calling the React agent that I created above with the question, what's the weather like in Tokyo? We'll see an example of this in the code session, but basically what this would do is give the model the prompt, the tools, and let it do its thing.

So a typical flow, I'm not going to spend too much time on this, but fairly logical constructor program which is decomposing your logic into the modules. You use your metrics to define basically the contours of how the program works and you optimize all that through um to to get your your uh your final result. So, another talk that this guy Chris Pototts just had maybe two days ago, um, where he made the point, this is what I was mentioning before, where Jeepa, which is, uh, you probably saw some of the the talks the other day, um, where the optimizers are on par or exceed the performance of something like GRPO, another kind of fine-tuning method. So, pretty impressive. I think it's an active area of research. people a lot smarter than me like Omar and Chris and others are are leading the way on this. But uh point being I think prompt op prompt optimization is a pretty exciting place to be and if nothing else is worth exploring. And [clears throat] then finally metrics again these are kind of the building blocks that allow you to define what success looks like for the optimizer. So this is what it's using and you can have many of these and we'll see examples of this where again at a high level your program works on inputs it works on outputs the optimizer is going to use the metrics to understand okay my last tweak in the prompt did it improve performance it did it degrade performance and the way you define your metrics uh provides that direct feedback for the optimizers to work on. Uh so here's another example, a super simple one from that time entry example I mentioned before. Um, so they can be the metrics can either be like fairly rigorous in terms of like does this equal one or or you know some type of equality check or a little bit more subjective where using LLM as a judge to say whatever was this generated um string does it adhere to these you know various criteria whatever it might be but that itself can be a metric and so all of this is to say it's a very long-winded way of saying in my opinion this is probably most if not all of what you need to construct arbitrarily complex workflows, data processing pipelines, business logic, whatever that might be. Different ways to work with LLMs. If nothing else, DSPI gives you the primitives that you need in order to build these modular composable systems. So, if you're interested in some people online, um there's many many more. There's a Discord community as well. Um, but usually these people are are on top of the latest and greatest and so would recommend giving them a follow. You don't need to follow me. I don't really do much, but uh the others on there are are really pretty good. Okay, so the fun part, we'll actually get into some to some code. So, if you haven't had a chance, now's your last chance to get the repo. U, but I'll just kind of go through a few different examples here of what we talked about. Maybe Yeah. Okay. Okay. So, I'll set up Phoenix, which is from Arise, uh, which is basically an obser an observability platform. Uh, I just did this today, so I don't know if it's going to work or not, but we'll we'll see. We'll give it a shot. Uh, but basically what this allows you to do is have a bunch of observability and tracing for all the calls that are happening under the hood. We'll see if this works. We'll give it like another 5 seconds. Um, but it should, I think, automatically do all this stuff for me. Yeah. So, let's see. Yeah. All right. So, something's up. Okay, cool. So, I'll just I'm just going to run through the notebook, which is a collection of different use cases, basically putting into practice a lot of what we just saw. Feel free to jump in any questions, anything like that. We'll start with this notebook. There's a couple of other uh more proper Python programs that we'll walk through afterwards. Uh but really the intent is a rapidfire review of different ways that DSPI has been useful to me and others. So load in the end file. Usually I'll have some type of config object like this where I can very easily use these later on. So if I'm like call like model mixing. So if I have like a super hairy problem or like some workload I know will need the power of a reasoning model like GPD5 or something else like that, I'll define multiple LM. So like one will be 41, one will be five, maybe I'll do a 41 nano um you know Gemini 2.5 flash, stuff like that. And then I can kind of intermingle or intersperse them depending on what I think or what I'm reasonably sure the workload will be. and you'll see how that comes into play in terms of classification and others. Um, I'll pull in a few others here. I'm I'm using open router for this. So, if you have an open router API key, would recommend plug plugging that in. So, now I have three different LLMs I can work with. I have Claude, I have Gemini, I have 41 mini. And then I'll ask basically for each of them who's best between Google Anthropic OpenAI. All of them are hedging a little bit. They say subjective, subjective, undefined. All right, great. It's not very helpful. But because DSPI works on Pyantic, I can define the answer as a literal. So I'm basically forcing it to only give me those three options and then I can go through each of those. And you can see each of them, of course, chooses their own organization. Um, the reason that those came back so fast is that DSP has caching automated under the hood. So as long as nothing has changed in terms of your uh your signature definitions or basically if nothing has changed this is super useful for testing it will just load it from the cache. Um so I ran this before that's why those came back so quickly. U but that's another kind of super useful um piece here. Let's see. Okay. Make sure we're up and running. So, if I change this to hello with a space, you can see we're making a live call. Okay, great. We're still up. So, super simple class sentiment classifier. Obviously, this can be built into something arbitrarily complex. Make this a little bit bigger. Um, but I'm basically I'm giving it the text, the sentiment that you saw before, and I'm adding that additional specification to say, okay, lower uh is more negative, higher is more positive. I'm going to define that as my signature. I'm going to pass this into just a super simple predict object. And then I'm going to say, okay, well, this hotel stinks. Okay, it's probably pretty negative. Now, if I flip that to I'm feeling pretty happy. Whoops. Good thing I'm not in a hotel right now. U you can see I'm feeling pretty happy. Comes down to eight. And this might not seem that impressive and you know it's it's not really but uh the the the important part here is that it just demonstrates the use of the shorthand um signature. So I have I have the string, I have the integer, I pass in the custom instructions which would be in the dock string if I use the class B classbased uh method. The other interesting part or or useful part about DSPI comes with a bunch of usage information built in. So um because it's cached, it's going to be an empty object. But when I change it, you can see that I'm using Azure right now, but for each call, you get this nice breakdown. and I think it's from late LLM, but allows you to very easily track your usage, token usage, etc. for observability and optimization and everything like that. Just nice little tidbits uh that are part of it here and there. Make this smaller. Uh we saw the example before in the slides, but I'm going to pull in that form 4 off of online. I'm going to create this doc objects using attachments. You can see some of the stuff it did under the hood. So, it pulled out um PDF plumber. It created markdown from it. Pulled out the images, etc. Again, I don't have to worry about all that. Attachments make that super easy. I'm going to show you what we're working with here. This case, we have the form four. And then I'm going to do that poor man's rag that I mentioned before. Okay, great. How many shares were were sold in total? It's going to go through that whole chain of thought and bring back the response. That's all well and good, but the power in my mind of DSPI is that you can have these arbitrarily complex data structures. That's fairly obvious because it uses paidantic and everything else, but you can get a little creative with it. So in this case, I'm going to say, okay, a different type of document analyzer signature. I'm just going to give it the document and then I'm just going to defer to the model on defining the structure of what it thinks is most important from the document. So in this case, [clears throat] I'm defining a dictionary object and so it will hopefully return to me a series of key value pairs that describe important information in the document in a structured way. And so you can see here again this is probably cached uh but I passed in I did it all in one line in this case but I'm saying I want to do chain of thought using the document analyzer signature and I'm going to pass in the input field which is just the document here. I'm going to pass in the document that I got before. And you can see here it pulled out bunch of great information in the super structured way. And I didn't have to really think about it. I just kind of deferred all this to the model to DSPI for how to do this. Now, of course, you can do the inverse in saying, okay, I have a very specific business use case. I have something specific in terms of the formatting or the content that I want to get out of the document. I define that as just kind of your typical paid classes. So in this case I want to pull out the if there's multiple transactions the schema itself important information like the filing date going to define the document analyzer schema signature. Uh again super simple input field which is just the document itself which is parsed by attachments gives me the text and the images and then I'm passing in the document schema parameter which has the document schema type which is defined above and this is the this is effectively what you would pass into structured outputs um but just doing it the DS pie where it's going to give you um basically the the output in that specific format. So you can see pulled out things super nicely. Filing date, form date, form type, transactions themselves, and then the ultimate answer. [clears throat] And it's nice because it exposes it in a way that you can use dot notation. So you can just very quickly access the the resulting objects. So looking at adapters, um I'll use another little tidbit from DSPI, which is the inspect history. So for those who want to know what's going on under the hood, inspect history will give you the raw dump of what's actually going on. So you can see here the system message that was uh constructed under the hood was all of this. So you can see input fields are document output fields or reasoning and the schema. It's going to pass these in. And then you can see here the actual document content that was extracted and put into the text and into the prompt uh with some metadata. This is all generated by attachments. And then you get the response which follows this specific format. So you can see the different fields that are here. And it's this kind of relatively arbitrary response um basically format for the for the names which is then parsed by the pie and passed back to you as the user. Um, so I can do okay response.document schema and get the the actual result. To show you what the BAML adapter looks like, we can basically do two different calls. So this is an example from uh my buddy Pashant uh online again. So what we do here is define pyantic model super simple one. Patient address and then patient details. Patient details has the patient address object within it. And then we're going to say we're going to create a super simple DSPI signature to say taking a clinical note which is a string. The patient info is the output type. And then note so I'm going to run this two different ways. The first time with the smart LLM that I mentioned before and just use the the built-in adapter. So I don't specify anything there. And then the second one will be using the BAML adapter which which is defined there. Um so I guess a few things going on here. One is the ability to use Python's uh context which is the the lines starting with with width which allow you to basically break out of what the global LLM um has been defined as and use a specific one just for that call. So you can see in this case I'm using the same LM but if I want to change this to like LM anthropic or something I think that should work. Um, but basically what that's doing is just offloading that call to the other whatever LLM that you're defining [clears throat] for that particular call and something happened. And I'm on a VPN, so let's kill that. Sorry, Alex Partners. Okay. Okay, great. So, we had two separate calls. One was to the smart LLM, which is I think 41. The other one was to Anthropic. Same. Everything else is the exact same. The notes exact same, etc. We got the same exact output. That's great. But what I wanted to show here is the adapters themselves. So in this case, I'm doing inspect history equals 2. So I'm going to get both of the last two calls. And we're going to see how the prompts are going to be different. And so you can see here the first one, this is the built-in JSON schema, this crazy long JSON string. Yeah, LLMs are good enough to to handle that, but um you know, probably not for super complicated ones. Um uh and then you see here for the the second one, it uses the BAML notation, which as we saw in the slides, a little bit easier to comprehend. Um and on super complicated use cases can actually have a measurable u improvement. Multimodal example, same sort of thing as before. I'll pull in the image itself. Let's just see what we're working with. Okay, great. We're looking at these various street signs. And I'm just going to ask it super simple question. It's this time of day. Can I park here now? When when should I leave? And you can see I'm just passing in again the super simple um shorthand for defining a signature which then I get out the the var the boolean in this case and a string of when I can leave. Um so modules themselves it's again fairly simple. You just kind of wrap all this in a class. Good question. >> So does it return reasoning by default always? >> Oh good question. Yeah. So when you do >> can you repeat the question? >> Yes. So for those online the question was does it always return reasoning by default? When you call DSPI.chain chain of thought as part of the module where it's built in. It's adding the reasoning u automatically into your response. So you're not defining that. It's a great question. It's not defined in the signature as you can see up here. Uh but it will add that in and expose that to you um to the extent that you want to retain it for any you know any reason. Uh but that's so if I ju if I changed this to predict you wouldn't get that same response, right? You just you literally just get that part. Um so that's actually a good segue to the modules. Um so module is basically just wrapping all that into some type of replic replicable uh logic. Um and so we're just we're giving it the signature here. We're saying selfpredict. We're in this case is just a demonstration of how it's being used as a class. So I'll just add this module identifier and sort some sort of counter but this can be any type of arbitrary business logic or control flow or any database action or whatever it might be. When this image analyzer class is called this function would run um and then when you actually invoke it this is when it's actually going to run the the core logic. And so you can see I'm just passing in the So I'm instantiating it the analyzer of AIE123 and then I'll call it. Great. It called that and you can see the counter incrementing each time I actually make the call. So super simple example. Um we don't have a ton of time but I'll I'll show you some of the other modules and how that kind of works out. Terms of tool calling fairly straightforward. I'm going to define two different functions perplexity search and get URL content. creating a bioagent module. So this is going to define Gemini 25 as this particular module's um LLM. It's going to create an answer generator object which is a react call. So I'm going to basically do tool calling whenever this is called and then the forward function is literally just calling that answer generator with the parameters that are provided to it. And then I'm creating an async version of that function as well. So I can do that here. I'm going to say okay identify instances where a particular person has been at their company for more than 10 years. It needs to do tool calling to do this to get the most up-to-date information. And so what this is doing and basically looping through um and it's going to call that bio agent which is using the tool calls in the background and it will make a determination as to whether their background is applicable per my criteria. In this case, Satia is true. Brian should be false. Uh but what's interesting here while that's going in it uh similar to the reasoning uh par or the reasoning object that you get back for chain of thought you can get a trajectory back for things like react. So you can see what tools it's calling the arguments that are passed in um and the observations for each of those calls which is nice for debugging and and other obviously other uses. Um I want to get to the other content so I'm going to speed through the rest of this. This is basically an async version of the same thing. So you would run both of them in parallel. Same idea. Um I'm going to skip the JEPA example here just for a second. Um I can show you what the output looks like, but basically what this is doing is creating a data set. It is showing you what's in the data set. It's creating a variety of signatures. In this case, it's going to create a system that categorizes and classifies different basically help messages um that is part of the data set. So, my sync is broken or my light is out or whatever it is. They want to classify whether it's positive, neutral, or negative and the uh the urgency of the actual message. It's going to categorize it and then it's going to pack all this stuff, all those different modules into a single support analyzer module. And then from there, what it's going to do is define a bunch of metrics which is based off of the data set itself. So it's going to say, okay, how do we score the urgency? This is a a very simple one where it's okay, it either matches or it doesn't. Um, and there's other ones where it can be a little bit more subjective and then you can run it. This going to take too long. Probably takes 20 minutes or so. Um but uh what it will do is basically evaluate the performance of the base model and then apply those metrics uh and iteratively come up with new prompts to uh to create that. Now I want to pause here just for a second because there's different types of metrics and in particular for Jeepa it uses feedback from the teacher model in this case. So it can work with the same level of model, but in particular when you're trying to use say a smaller model, um it can actually provide textual feedback. So, it says not only did you get this classification wrong, but it's going to give you some additional um information or feedback as you can see here for why it got it wrong or what the answer should have been, which allows it you you can read the paper, but it basically allows it to um iteratively find that kind of paro frontier of how it should uh tweak the prompt to optimize it based off that feedback. It basically just tightens that iteration loop. Um you can see there's a bunch here. Um and then you can run it and see how it works. [snorts] Um but kind of just to give you a concrete example of how it all comes together. So we took a bunch of those examples from before. We're basically basically going to do a bit of um categorization. So I have things like contracts, I have images, I have different things that one DSPI program can comprehend and do some type of processing with. So this is something that we see fairly regularly in terms of we might run into a client situation where they have just a big dump of of files. They don't really know what's in it. They want to find something of uh they want to maybe find SEC filings and process them a certain way. they want to find contracts and process those a certain way. Maybe there's some images in in there and they want to process those a certain way. Uh [snorts] so this is an example of how you would do that where if I start at the bottom here, this is a regular Python file. Um and it uses DSPI to do all those things I just mentioned. So we're pulling in the configurations, we're setting the regular LM, the small and one we use for an image. As an example, Gemini might Gemini models might be better at image recognition than others. So I might want to defer or use a particular model for a particular workload. So if I detect an image, I will route the request to Gemini. If I detect something else, I'll route it to a 4.1 or whatever it might be. So I'm going to process a single file. And what it does is use our handy attachments um library to put it into a format that we can use. And then I'm going to classify it. And it's not super obvious here, but I'm getting a file type from this classify file uh function call. And then I'm doing some different type of logic depending on what type of file it is. So if it's an SEC filing, I do certain things. If it's a certain type of SEC filing, I do something else. Uh, if [snorts] it's a contract, maybe I'll summarize it. If it's something that looks like city infrastructure, in this case, the image that we saw before, I might do some more visual interpretation of it. Um, so if I dive into classify file super quick, it's running the document classifier. And all that is is basically doing a predict on the image from the file. and um making sure it returns a type. Where is this returns a type which would be document type and so you can see here at the end of the day it's a fairly simple signature and so what we've done is basically take the PDF file in this case take all the images from it and take the first image or first few images in this case a list of images as the input field and I'm saying okay just give me the type what is this and I'm giving it an option of these document types so obviously say this is a fairly simple use case but it's basically saying given these three images the first three pages of a document is it an SEC filing is it a patent filing is the contract city infrastructure pretty different things so the model really shouldn't have an issue with any of those and then we have a catchall bucket for other and then as I mentioned before um depending on the file type that you get back you can process them differently so I'm using the small model to do the same type of form4 extraction that we saw before um and then asserting basically in this case that it is what we think it is. Um a contract in this case we're saying uh let's see I have like 10 more minutes so we can go we'll we'll stop after this uh up to this file but for the particular contract we'll go we'll create this summarizer object. So we'll go through as many pages as there are. We'll do some uh basically recursive summarization of that using a separate DSPI function and then we'll detect some type of boundaries of that document too. So we'll say I want the summaries and I want the boundaries of the document. Um and then we'll print those things out. So let's just see if I can run this. It's going to classify it should as a [clears throat] contract. >> So is you're just relying on the model itself to realize that it's a city infrastructure. >> Yeah. The question was I'm I'm just relying on the model to determine if it's a city infrastructure. Yes. I mean this is more just like a workshop quick and dirty example. It's only because there's one picture of the street signs. Um, and if we look in the data folder, I have a contract, some image that's irrelevant, the form for SEC filing, and then the parking too. Um, they're pretty different. The model should have no problem out of those categories that I gave it to categorize it properly. In some type of production use case, you would want much more stringent or maybe even multiple passes of classification, maybe using different models to do that. Um but yeah, given those options, at least the many times I've run it, had no problem. So in this case, I gave it um one of these contract documents and it ran some additional summarization logic under the hood. So, if I go to that super quick, um you can find all this in the code, but basically what it does is use three separate signatures to basically decompose the contents of the the um the contract and then summarize them up. So, it's basically just iteratively working through each of the chunks of the document to create a summary that you see here at the bottom. And then just for good measure, we're also detecting basically the the boundaries of the document to say, okay, here's out of the 13 pages, you have the main document and then some of the exhibits or the schedules that are a part of it. So, let me just bring it up super quick just to show you what we're working with. This is just some random thing I found online. And you can see so it said the main document was from page 0 to six and the way and so we zero 1 2 3 4 5 six seems reasonable. Now we have the start of schedule one. Schedule one it says it's the next two pages. That looks pretty good. Schedule two is just the one page 9 to9. That looks good. and then schedule three through to the end of the document. And that looks pretty good, too. And so the way we did that under the hood was basically take the PDF, convert it to a list of images and then for each of the images pass those to classifier um and then use that to well let's just look at the code but basically take the list of those classifications give that to another DSPI signature to say given these classifications of the document give me the structure and basically give me a key pair of you name of the section and two integers, a tupole of integers that detect or that uh determine the um you know the boundaries essentially. Um so that's what that part does. Um [clears throat] if we go back so city infrastructure, I'll do this one super quick just because it's pretty interesting on how it uses tool calls. And while this is running, I should use the right one. Hold on. >> [clears throat] >> Yeah, >> good question. The second part like when you generated the list of like my documents from 0 to six, did you have like original document as an input or no? >> No. Uh so let let's just go to that uh that was super quick. So that should be boundary detector. So, there's a blog post on this that I published probably in August or so that goes into a little bit more detail. The code is actually pretty crappy in that one. It's it's going to be better here. Um, but basically what it does is this is probably the main logic. So, for each of the images in the PDF, we're going to call classify page. We're going to gather the results. So it's doing all that asynchronously pulling it all back saying okay all these you know all the different page classifications that there are and then I pass the output of that into a new signature that says given tupil of p I don't even define it here given tupil of page and classification give me this I don't know relatively complicated output of a dictionary of a string tupil integer integer and I give it this set of instructions to say just detect the boundaries. Like this is all very like non-production code obviously, but the point is that you can do these types of things super super quickly. Like I'm not specifying much not giving it much context and it worked like pretty well. Like it it's worked pretty well in most of my testing. Now obviously there is a ton of low hanging fruit in terms of ways to improve that, optimize it, etc. Um, but all this is doing is taking that signature, these instructions, and then I call react. And then all I give it is, uh, the ability to basically self-reflect and call um, get page images. So, it says, okay, I'm going to look at this boundary. Well, let me get the the page images for these three pages to and make sure basically that the boundary is correct. And then it uses that to construct the final answer. And so it's really this is a perfect example of like the tight iteration loop that you can have both in um building it but then the you can kind of take advantage of the model's introspective ability if you will to use function calls against the data itself the data it generated itself etc to kind of keep that loop going. question. >> So under the hood, the the beauty of ESP then is that it enforces kind of structured output on a on a model. >> I mean yes, I think that's probably reductive of of like its full potential, but generally that's that's correct. I mean yes, you can use structured outputs, but you have to do a bunch of crap basically to coordinate like feeding all the feeding that into the rest of the program. maybe you want to call a model differently or use XML here or use a different type of model or whatever it might be um to to do that. So absolutely yeah I'm not saying this is the only way obviously to kind of create these applications or that you shouldn't use Pantic or shouldn't use structured outputs. You absolutely should. Um, it's just a way that once you kind of wrap your head around the the primitives that DSPI gives you, you can start to very quickly build these types of arguably uh I mean these are like prototypes right now, but like if you want to take this to the next level to production scale, you have all the pieces in front of you to be able to do that. Um, any other questions? I probably got about five minutes left. Go ahead. Can you talk about your experience using optimization and just >> Yeah. Yeah. So Jeep and actually I'll pull up uh I I ran one right before this. Um this uses a a different algorithm called my row but basically um the optimizers as long as you have well structured data. So for the machine learning folks in the room, which is probably everybody, obviously the quality of your of your data is very important, um you don't need thousands and thousands of examples necessarily, but as long as you have enough, maybe 10 to 100 of inputs and outputs. [clears throat] And if you're constructing your metrics in a way that is relatively intuitive and and that, you know, accurately describes what you're trying to achieve, the improvement can be pretty significant. Um, and so that time entry corrector thing that I mentioned before, uh, you can see the output of here. It's kind of iterating through. It's measuring the output metrics for each of these. And then you can see all the way at the bottom once it goes through all of its optimization stuff. You can see the actual performance on um, the basic versus the optimized model. In this case, went from 86 to 89. And then interestingly, this is still in development, this one in particular, but you can break it down by metric. So you can see where the model's optimizing better, performing better across certain metrics. And this can be really telling as to whether you need to tweak your metric, maybe you need to decompose your metric, maybe there's other areas within your data set, or the the basically the structure of your program that you can improve. Um, but it's a really nice way to understand what's going under the under the hood. And if if you don't care about some of these and the optimizer isn't doing as well on them, maybe you can maybe you can throw them out, too. So, it's it's a very kind of flexible system, flexible way of kind of doing all that. >> Yeah. What's the output of the optimization? Like what do you get out of it and then how do you use that object, whatever it is? >> Yeah. Yeah. So the output of the optimizers is basically just another um it's almost like a compiled object if you will. >> So DSPI allows you to save and load programs as well. So the output of the optimizer is basically just a module that you can then serialize and save off somewhere >> or you can call it later uh as you would any other module >> and it's just manipulating the phrasing of the prompt. So like what is it actually like you know what's its solution space look like? >> Yeah. Yeah. under the hood, it's literally just iterating on the actual prompt itself. Maybe it's adding additional instructions. It's saying, "Well, I keep failing on this particular thing, like not capitalizing the names correctly. I need to add [clears throat] in my upfront criteria in the prompt an instruction to the model to say you must capitalize names properly." And Chris uh who I mentioned before has a really good way of putting this and I'm going to butcher it now, but like the optimizer is basically finding latent requirements that you might not have specified initially up front, but based off of the data, it's kind of like a poor man's deep learning, I guess, but like it's learning from the data. It's learning what it's doing well, what what it's doing not so well, and it's dynamically constructing a prompt that improves the performance based off of your metrics. And is that like LMG guided like is it like about like capitalization? >> Yeah. Yeah. Question being is it all LLM guided? Yes. Particularly for Jeepa it's using LLM to improve LLM's performance. So it's using the LLM to dynamically construct new prompts which are then fed into the system measured and then it kind of iterates. So it's using AI to build AI if you will. >> Thank you. >> Yeah question. Why is the solution object not just the optimized prompt? >> Why is the solution object not what? >> Not just the optimized prompt. Why are you using >> Oh, absolutely is. You can get it under the hood. I mean, you can The question was why don't you just get the optimized prompt? You can absolutely. Um, >> what else is there besides >> the the So, what else is there other than the prompt? The DSPI object itself. So the module the way things um well we can probably look at one if we have time. Um >> if I could see a dump of what gets you know what is the optimized state that would be interesting. >> Yeah. Yeah sure. Let me see if I can find one quick. Um but fundamentally at the end of the day yes you get an optimized prompt a string that you can dump somewhere if you if you want to. Um actually um >> there's a lot of pieces to the signature, right? So it's like how you describe your feels in the doc. >> This is a perfect segue and I'll I'll conclude right after this. I was playing around with something I was well I was playing around this thing called DSPIHub that I kind of created to create a repository of optimized programs. So basically like if you're an expert in whatever you optimize an LLM against this data set or have a great classifier for city infrastructure images or whatever kind of like a hugging face you can download something that has been pre-optimized and then what I have here this is the actual loaded program this would be the output of the optimized process or it it is and then I can call it as I would any anything else and so you you can see here this is the output and I used the optimized program that I downloaded from from this hub. And if we inspect maybe the loaded program, you can see under the hood, it's a predict object with a string signature of time and reasoning. Here is the optimized prompt. Ultimately, this is the output of the optimization process. this long string here. Um, and then the various uh specifications and definitions of the inputs and outputs. >> Have you found specific uses of those? Like to his question like what is it? What can you do with that? >> It's up to your it's up to your use case. So if I if I have a so a document classifier might be a good example. If in my business I come across whatever documents of a certain type, I might optimize a classifier against those and then I can use that somewhere else on a different project or something like that. So out of 100,000 documents, I want to find only the pages that have an invoice on it as an example. Now sure 100% you can use a typical ML classifier to do that. That's great. This is just an example. We can also theoretically train or optimize a model to do that type of classification or some type of generation of text or what have you which then you have the optimized state of which then lives in your data processing pipeline you know and you can use it for other types of purposes or give it to other teams or whatever it might be. So it's just up to your particular use case. um something like this like hub who maybe it's not useful because each individual's use case is so hyper specific I don't really know but um yeah you can do with it kind of whatever you want last question yeah >> is generally you know like using DSP something where people kind of do replays just to optimize their prop or is there a way to sort of do it in real time given delays What I mean by delayed is okay chat GPT gives you your answer and you can thumbs up or thumbs down. You know that thumbs up comes you know 10 minutes later, 30 minutes later, a day later, right? >> So is the question more about like continuous learning like how would you do that here? >> You can be the judge. >> Well, how are you feeding back delayed metrics to optimize it? Why why would it need to be delayed? Because you know usually the feedback is from the user, right? Like delayed. >> Yeah. Well, then >> yeah, that's right. You it basically be added to the data set and then you would use the latest optimize and just keep keep optimizing off of that >> ground truth data set. >> That's right. >> You will collect the outputs of your optimization and feed it back and the loop hits. >> Yeah. But that Why you're trying to do offline optimization, right? >> Yes. >> But I'm I'm asking, can you do this online where with the metric feedback? >> If you're good if you're a good enough engineer, you probably do it. But >> I'm not I'm not recommending replacing ML models with like optimized DSPI programs for particular use cases. Maybe like classification is a terrible example. I recognize that. But for other other are other in theory, yes, you know, you could do something like that. Yes. But for for particular LLM tasks, I'm sure we all have interesting ones. If you have something that is relatively well defined where you have known inputs and outputs, it might be a candidate for something worth optimizing. If nothing else, to transfer it to a smaller model to preserve the level of performance at a lower cost. That's really one of the biggest benefits I see. All right, last last question. I've heard that uh DSPI is can be kind of expensive because you're doing all these LM calls. >> Um so I was curious your experience with that and maybe relatedly like if you have any experience with like large context in your optimization data set ways of shrinking those. >> Yeah. So the question was do can BSI be expensive and then for large context kind of how have you seen that? How have you managed that? The expensive part is totally up to you. If you call a function a million times asynchronously, you're going to generate a lot of cost. I don't think DSPI necessarily maybe it makes it easier to call things, but it's not inherently expensive. It might, to your point, add more content to the prompt. Like, sure, the signature is a string, but the actual text that's sent to the model is much longer than that. That's totally true. I wouldn't say that it's a large cost driver. I mean it again it's ultimately it's more more of a programming paradigm. So you can write your compressed adapter if you want that like you know reduces the amount that's sent to the to uh to the model. Um in terms of large context I it's kind of the same answer I think in terms of if you're worried about that maybe you have some additional logic either in the program itself or in an adapter or part of the module that keeps track of that. Maybe you do some like context compression or something like that. There's some really good talks about that past few days. Obviously, I have a feeling that that will kind of go away at some point where either context windows get bigger or context management is abstracted away somehow. I don't really have an answer just that's more of an intuition. Um, but DSP again kind of gives you the tools, the primitives for you to do that should you choose. Um, and kind of track that state, track that management over time. So, I think that's it. We're going to get kicked out soon. So, thanks so much for your time. Really appreciate it. [music] [music]

Um I want to get to the other content so I'm going to speed through the rest of this. This is basically an async version of the same thing. So you would run both of them in parallel. Same idea. Um I'm going to skip the JEPA example here just for a second. Um I can show you what the output looks like, but basically what this is doing is creating a data set. It is showing you what's in the data set. It's creating a variety of signatures. In this case, it's going to create a system that categorizes and classifies different basically help messages um that is part of the data set. So, my sync is broken or my light is out or whatever it is. They want to classify whether it's positive, neutral, or negative and the uh the urgency of the actual message. It's going to categorize it and then it's going to pack all this stuff, all those different modules into a single support analyzer module. And then from there, what it's going to do is define a bunch of metrics which is based off of the data set itself. So it's going to say, okay, how do we score the urgency? This is a a very simple one where it's okay, it either matches or it doesn't. Um, and there's other ones where it can be a little bit more subjective and then you can run it. This going to take too long. Probably takes 20 minutes or so. Um but uh what it will do is basically evaluate the performance of the base model and then apply those metrics uh and iteratively come up with new prompts to uh to create that. Now I want to pause here just for a second because there's different types of metrics and in particular for Jeepa it uses feedback from the teacher model in this case. So it can work with the same level of model, but in particular when you're trying to use say a smaller model, um it can actually provide textual feedback. So, it says not only did you get this classification wrong, but it's going to give you some additional um information or feedback as you can see here for why it got it wrong or what the answer should have been, which allows it you you can read the paper, but it basically allows it to um iteratively find that kind of paro frontier of how it should uh tweak the prompt to optimize it based off that feedback. It basically just tightens that iteration loop. Um you can see there's a bunch here. Um and then you can run it and see how it works. [snorts] Um but kind of just to give you a concrete example of how it all comes together. So we took a bunch of those examples from before. We're basically basically going to do a bit of um categorization. So I have things like contracts, I have images, I have different things that one DSPI program can comprehend and do some type of processing with. So this is something that we see fairly regularly in terms of we might run into a client situation where they have just a big dump of of files. They don't really know what's in it. They want to find something of uh they want to maybe find SEC filings and process them a certain way. they want to find contracts and process those a certain way. Maybe there's some images in in there and they want to process those a certain way. Uh [snorts] so this is an example of how you would do that where if I start at the bottom here, this is a regular Python file. Um and it uses DSPI to do all those things I just mentioned. So we're pulling in the configurations, we're setting the regular LM, the small and one we use for an image. As an example, Gemini might Gemini models might be better at image recognition than others. So I might want to defer or use a particular model for a particular workload. So if I detect an image, I will route the request to Gemini. If I detect something else, I'll route it to a 4.1 or whatever it might be. So I'm going to process a single file. And what it does is use our handy attachments um library to put it into a format that we can use. And then I'm going to classify it. And it's not super obvious here, but I'm getting a file type from this classify file uh function call. And then I'm doing some different type of logic depending on what type of file it is. So if it's an SEC filing, I do certain things. If it's a certain type of SEC filing, I do something else. Uh, if [snorts] it's a contract, maybe I'll summarize it. If it's something that looks like city infrastructure, in this case, the image that we saw before, I might do some more visual interpretation of it. Um, so if I dive into classify file super quick, it's running the document classifier. And all that is is basically doing a predict on the image from the file. and um making sure it returns a type. Where is this returns a type which would be document type and so you can see here at the end of the day it's a fairly simple signature and so what we've done is basically take the PDF file in this case take all the images from it and take the first image or first few images in this case a list of images as the input field and I'm saying okay just give me the type what is this and I'm giving it an option of these document types so obviously say this is a fairly simple use case but it's basically saying given these three images the first three pages of a document is it an SEC filing is it a patent filing is the contract city infrastructure pretty different things so the model really shouldn't have an issue with any of those and then we have a catchall bucket for other and then as I mentioned before um depending on the file type that you get back you can process them differently so I'm using the small model to do the same type of form4 extraction that we saw before um and then asserting basically in this case that it is what we think it is. Um a contract in this case we're saying uh let's see I have like 10 more minutes so we can go we'll we'll stop after this uh up to this file but for the particular contract we'll go we'll create this summarizer object. So we'll go through as many pages as there are. We'll do some uh basically recursive summarization of that using a separate DSPI function and then we'll detect some type of boundaries of that document too. So we'll say I want the summaries and I want the boundaries of the document. Um and then we'll print those things out. So let's just see if I can run this. It's going to classify it should as a [clears throat] contract. >> So is you're just relying on the model itself to realize that it's a city infrastructure. >> Yeah. The question was I'm I'm just relying on the model to determine if it's a city infrastructure. Yes. I mean this is more just like a workshop quick and dirty example. It's only because there's one picture of the street signs. Um, and if we look in the data folder, I have a contract, some image that's irrelevant, the form for SEC filing, and then the parking too. Um, they're pretty different. The model should have no problem out of those categories that I gave it to categorize it properly. In some type of production use case, you would want much more stringent or maybe even multiple passes of classification, maybe using different models to do that. Um but yeah, given those options, at least the many times I've run it, had no problem. So in this case, I gave it um one of these contract documents and it ran some additional summarization logic under the hood. So, if I go to that super quick, um you can find all this in the code, but basically what it does is use three separate signatures to basically decompose the contents of the the um the contract and then summarize them up. So, it's basically just iteratively working through each of the chunks of the document to create a summary that you see here at the bottom. And then just for good measure, we're also detecting basically the the boundaries of the document to say, okay, here's out of the 13 pages, you have the main document and then some of the exhibits or the schedules that are a part of it. So, let me just bring it up super quick just to show you what we're working with. This is just some random thing I found online. And you can see so it said the main document was from page 0 to six and the way and so we zero 1 2 3 4 5 six seems reasonable. Now we have the start of schedule one. Schedule one it says it's the next two pages. That looks pretty good. Schedule two is just the one page 9 to9. That looks good. and then schedule three through to the end of the document. And that looks pretty good, too. And so the way we did that under the hood was basically take the PDF, convert it to a list of images and then for each of the images pass those to classifier um and then use that to well let's just look at the code but basically take the list of those classifications give that to another DSPI signature to say given these classifications of the document give me the structure and basically give me a key pair of you name of the section and two integers, a tupole of integers that detect or that uh determine the um you know the boundaries essentially. Um so that's what that part does. Um [clears throat] if we go back so city infrastructure, I'll do this one super quick just because it's pretty interesting on how it uses tool calls. And while this is running, I should use the right one. Hold on. >> [clears throat] >> Yeah, >> good question. The second part like when you generated the list of like my documents from 0 to six, did you have like original document as an input or no? >> No. Uh so let let's just go to that uh that was super quick. So that should be boundary detector. So, there's a blog post on this that I published probably in August or so that goes into a little bit more detail. The code is actually pretty crappy in that one. It's it's going to be better here. Um, but basically what it does is this is probably the main logic. So, for each of the images in the PDF, we're going to call classify page. We're going to gather the results. So it's doing all that asynchronously pulling it all back saying okay all these you know all the different page classifications that there are and then I pass the output of that into a new signature that says given tupil of p I don't even define it here given tupil of page and classification give me this I don't know relatively complicated output of a dictionary of a string tupil integer integer and I give it this set of instructions to say just detect the boundaries. Like this is all very like non-production code obviously, but the point is that you can do these types of things super super quickly. Like I'm not specifying much not giving it much context and it worked like pretty well. Like it it's worked pretty well in most of my testing. Now obviously there is a ton of low hanging fruit in terms of ways to improve that, optimize it, etc. Um, but all this is doing is taking that signature, these instructions, and then I call react. And then all I give it is, uh, the ability to basically self-reflect and call um, get page images. So, it says, okay, I'm going to look at this boundary. Well, let me get the the page images for these three pages to and make sure basically that the boundary is correct. And then it uses that to construct the final answer. And so it's really this is a perfect example of like the tight iteration loop that you can have both in um building it but then the you can kind of take advantage of the model's introspective ability if you will to use function calls against the data itself the data it generated itself etc to kind of keep that loop going. question. >> So under the hood, the the beauty of ESP then is that it enforces kind of structured output on a on a model. >> I mean yes, I think that's probably reductive of of like its full potential, but generally that's that's correct. I mean yes, you can use structured outputs, but you have to do a bunch of crap basically to coordinate like feeding all the feeding that into the rest of the program. maybe you want to call a model differently or use XML here or use a different type of model or whatever it might be um to to do that. So absolutely yeah I'm not saying this is the only way obviously to kind of create these applications or that you shouldn't use Pantic or shouldn't use structured outputs. You absolutely should. Um, it's just a way that once you kind of wrap your head around the the primitives that DSPI gives you, you can start to very quickly build these types of arguably uh I mean these are like prototypes right now, but like if you want to take this to the next level to production scale, you have all the pieces in front of you to be able to do that. Um, any other questions? I probably got about five minutes left. Go ahead. Can you talk about your experience using optimization and just >> Yeah. Yeah. So Jeep and actually I'll pull up uh I I ran one right before this. Um this uses a a different algorithm called my row but basically um the optimizers as long as you have well structured data. So for the machine learning folks in the room, which is probably everybody, obviously the quality of your of your data is very important, um you don't need thousands and thousands of examples necessarily, but as long as you have enough, maybe 10 to 100 of inputs and outputs. [clears throat] And if you're constructing your metrics in a way that is relatively intuitive and and that, you know, accurately describes what you're trying to achieve, the improvement can be pretty significant. Um, and so that time entry corrector thing that I mentioned before, uh, you can see the output of here. It's kind of iterating through. It's measuring the output metrics for each of these. And then you can see all the way at the bottom once it goes through all of its optimization stuff. You can see the actual performance on um, the basic versus the optimized model. In this case, went from 86 to 89. And then interestingly, this is still in development, this one in particular, but you can break it down by metric. So you can see where the model's optimizing better, performing better across certain metrics. And this can be really telling as to whether you need to tweak your metric, maybe you need to decompose your metric, maybe there's other areas within your data set, or the the basically the structure of your program that you can improve. Um, but it's a really nice way to understand what's going under the under the hood. And if if you don't care about some of these and the optimizer isn't doing as well on them, maybe you can maybe you can throw them out, too. So, it's it's a very kind of flexible system, flexible way of kind of doing all that. >> Yeah. What's the output of the optimization? Like what do you get out of it and then how do you use that object, whatever it is? >> Yeah. Yeah. So the output of the optimizers is basically just another um it's almost like a compiled object if you will. >> So DSPI allows you to save and load programs as well. So the output of the optimizer is basically just a module that you can then serialize and save off somewhere >> or you can call it later uh as you would any other module >> and it's just manipulating the phrasing of the prompt. So like what is it actually like you know what's its solution space look like? >> Yeah. Yeah. under the hood, it's literally just iterating on the actual prompt itself. Maybe it's adding additional instructions. It's saying, "Well, I keep failing on this particular thing, like not capitalizing the names correctly. I need to add [clears throat] in my upfront criteria in the prompt an instruction to the model to say you must capitalize names properly." And Chris uh who I mentioned before has a really good way of putting this and I'm going to butcher it now, but like the optimizer is basically finding latent requirements that you might not have specified initially up front, but based off of the data, it's kind of like a poor man's deep learning, I guess, but like it's learning from the data. It's learning what it's doing well, what what it's doing not so well, and it's dynamically constructing a prompt that improves the performance based off of your metrics. And is that like LMG guided like is it like about like capitalization? >> Yeah. Yeah. Question being is it all LLM guided? Yes. Particularly for Jeepa it's using LLM to improve LLM's performance. So it's using the LLM to dynamically construct new prompts which are then fed into the system measured and then it kind of iterates. So it's using AI to build AI if you will. >> Thank you. >> Yeah question. Why is the solution object not just the optimized prompt? >> Why is the solution object not what? >> Not just the optimized prompt. Why are you using >> Oh, absolutely is. You can get it under the hood. I mean, you can The question was why don't you just get the optimized prompt? You can absolutely. Um, >> what else is there besides >> the the So, what else is there other than the prompt? The DSPI object itself. So the module the way things um well we can probably look at one if we have time. Um >> if I could see a dump of what gets you know what is the optimized state that would be interesting. >> Yeah. Yeah sure. Let me see if I can find one quick. Um but fundamentally at the end of the day yes you get an optimized prompt a string that you can dump somewhere if you if you want to. Um actually um >> there's a lot of pieces to the signature, right? So it's like how you describe your feels in the doc. >> This is a perfect segue and I'll I'll conclude right after this. I was playing around with something I was well I was playing around this thing called DSPIHub that I kind of created to create a repository of optimized programs. So basically like if you're an expert in whatever you optimize an LLM against this data set or have a great classifier for city infrastructure images or whatever kind of like a hugging face you can download something that has been pre-optimized and then what I have here this is the actual loaded program this would be the output of the optimized process or it it is and then I can call it as I would any anything else and so you you can see here this is the output and I used the optimized program that I downloaded from from this hub. And if we inspect maybe the loaded program, you can see under the hood, it's a predict object with a string signature of time and reasoning. Here is the optimized prompt. Ultimately, this is the output of the optimization process. this long string here. Um, and then the various uh specifications and definitions of the inputs and outputs. >> Have you found specific uses of those? Like to his question like what is it? What can you do with that? >> It's up to your it's up to your use case. So if I if I have a so a document classifier might be a good example. If in my business I come across whatever documents of a certain type, I might optimize a classifier against those and then I can use that somewhere else on a different project or something like that. So out of 100,000 documents, I want to find only the pages that have an invoice on it as an example. Now sure 100% you can use a typical ML classifier to do that. That's great. This is just an example. We can also theoretically train or optimize a model to do that type of classification or some type of generation of text or what have you which then you have the optimized state of which then lives in your data processing pipeline you know and you can use it for other types of purposes or give it to other teams or whatever it might be. So it's just up to your particular use case. um something like this like hub who maybe it's not useful because each individual's use case is so hyper specific I don't really know but um yeah you can do with it kind of whatever you want last question yeah >> is generally you know like using DSP something where people kind of do replays just to optimize their prop or is there a way to sort of do it in real time given delays What I mean by delayed is okay chat GPT gives you your answer and you can thumbs up or thumbs down. You know that thumbs up comes you know 10 minutes later, 30 minutes later, a day later, right? >> So is the question more about like continuous learning like how would you do that here? >> You can be the judge. >> Well, how are you feeding back delayed metrics to optimize it? Why why would it need to be delayed? Because you know usually the feedback is from the user, right? Like delayed. >> Yeah. Well, then >> yeah, that's right. You it basically be added to the data set and then you would use the latest optimize and just keep keep optimizing off of that >> ground truth data set. >> That's right. >> You will collect the outputs of your optimization and feed it back and the loop hits. >> Yeah. But that Why you're trying to do offline optimization, right? >> Yes. >> But I'm I'm asking, can you do this online where with the metric feedback? >> If you're good if you're a good enough engineer, you probably do it. But >> I'm not I'm not recommending replacing ML models with like optimized DSPI programs for particular use cases. Maybe like classification is a terrible example. I recognize that. But for other other are other in theory, yes, you know, you could do something like that. Yes. But for for particular LLM tasks, I'm sure we all have interesting ones. If you have something that is relatively well defined where you have known inputs and outputs, it might be a candidate for something worth optimizing. If nothing else, to transfer it to a smaller model to preserve the level of performance at a lower cost. That's really one of the biggest benefits I see. All right, last last question. I've heard that uh DSPI is can be kind of expensive because you're doing all these LM calls. >> Um so I was curious your experience with that and maybe relatedly like if you have any experience with like large context in your optimization data set ways of shrinking those. >> Yeah. So the question was do can BSI be expensive and then for large context kind of how have you seen that? How have you managed that? The expensive part is totally up to you. If you call a function a million times asynchronously, you're going to generate a lot of cost. I don't think DSPI necessarily maybe it makes it easier to call things, but it's not inherently expensive. It might, to your point, add more content to the prompt. Like, sure, the signature is a string, but the actual text that's sent to the model is much longer than that. That's totally true. I wouldn't say that it's a large cost driver. I mean it again it's ultimately it's more more of a programming paradigm. So you can write your compressed adapter if you want that like you know reduces the amount that's sent to the to uh to the model. Um in terms of large context I it's kind of the same answer I think in terms of if you're worried about that maybe you have some additional logic either in the program itself or in an adapter or part of the module that keeps track of that. Maybe you do some like context compression or something like that. There's some really good talks about that past few days. Obviously, I have a feeling that that will kind of go away at some point where either context windows get bigger or context management is abstracted away somehow. I don't really have an answer just that's more of an intuition. Um, but DSP again kind of gives you the tools, the primitives for you to do that should you choose. Um, and kind of track that state, track that management over time. So, I think that's it. We're going to get kicked out soon. So, thanks so much for your time. Really appreciate it. [music] [music]

Yeah. What's the output of the optimization? Like what do you get out of it and then how do you use that object, whatever it is? >> Yeah. Yeah. So the output of the optimizers is basically just another um it's almost like a compiled object if you will. >> So DSPI allows you to save and load programs as well. So the output of the optimizer is basically just a module that you can then serialize and save off somewhere >> or you can call it later uh as you would any other module >> and it's just manipulating the phrasing of the prompt. So like what is it actually like you know what's its solution space look like? >> Yeah. Yeah. under the hood, it's literally just iterating on the actual prompt itself. Maybe it's adding additional instructions. It's saying, "Well, I keep failing on this particular thing, like not capitalizing the names correctly. I need to add [clears throat] in my upfront criteria in the prompt an instruction to the model to say you must capitalize names properly." And Chris uh who I mentioned before has a really good way of putting this and I'm going to butcher it now, but like the optimizer is basically finding latent requirements that you might not have specified initially up front, but based off of the data, it's kind of like a poor man's deep learning, I guess, but like it's learning from the data. It's learning what it's doing well, what what it's doing not so well, and it's dynamically constructing a prompt that improves the performance based off of your metrics. And is that like LMG guided like is it like about like capitalization? >> Yeah. Yeah. Question being is it all LLM guided? Yes. Particularly for Jeepa it's using LLM to improve LLM's performance. So it's using the LLM to dynamically construct new prompts which are then fed into the system measured and then it kind of iterates. So it's using AI to build AI if you will. >> Thank you. >> Yeah question. Why is the solution object not just the optimized prompt? >> Why is the solution object not what? >> Not just the optimized prompt. Why are you using >> Oh, absolutely is. You can get it under the hood. I mean, you can The question was why don't you just get the optimized prompt? You can absolutely. Um, >> what else is there besides >> the the So, what else is there other than the prompt? The DSPI object itself. So the module the way things um well we can probably look at one if we have time. Um >> if I could see a dump of what gets you know what is the optimized state that would be interesting. >> Yeah. Yeah sure. Let me see if I can find one quick. Um but fundamentally at the end of the day yes you get an optimized prompt a string that you can dump somewhere if you if you want to. Um actually um >> there's a lot of pieces to the signature, right? So it's like how you describe your feels in the doc. >> This is a perfect segue and I'll I'll conclude right after this. I was playing around with something I was well I was playing around this thing called DSPIHub that I kind of created to create a repository of optimized programs. So basically like if you're an expert in whatever you optimize an LLM against this data set or have a great classifier for city infrastructure images or whatever kind of like a hugging face you can download something that has been pre-optimized and then what I have here this is the actual loaded program this would be the output of the optimized process or it it is and then I can call it as I would any anything else and so you you can see here this is the output and I used the optimized program that I downloaded from from this hub. And if we inspect maybe the loaded program, you can see under the hood, it's a predict object with a string signature of time and reasoning. Here is the optimized prompt. Ultimately, this is the output of the optimization process. this long string here. Um, and then the various uh specifications and definitions of the inputs and outputs. >> Have you found specific uses of those? Like to his question like what is it? What can you do with that? >> It's up to your it's up to your use case. So if I if I have a so a document classifier might be a good example. If in my business I come across whatever documents of a certain type, I might optimize a classifier against those and then I can use that somewhere else on a different project or something like that. So out of 100,000 documents, I want to find only the pages that have an invoice on it as an example. Now sure 100% you can use a typical ML classifier to do that. That's great. This is just an example. We can also theoretically train or optimize a model to do that type of classification or some type of generation of text or what have you which then you have the optimized state of which then lives in your data processing pipeline you know and you can use it for other types of purposes or give it to other teams or whatever it might be. So it's just up to your particular use case. um something like this like hub who maybe it's not useful because each individual's use case is so hyper specific I don't really know but um yeah you can do with it kind of whatever you want last question yeah >> is generally you know like using DSP something where people kind of do replays just to optimize their prop or is there a way to sort of do it in real time given delays What I mean by delayed is okay chat GPT gives you your answer and you can thumbs up or thumbs down. You know that thumbs up comes you know 10 minutes later, 30 minutes later, a day later, right? >> So is the question more about like continuous learning like how would you do that here? >> You can be the judge. >> Well, how are you feeding back delayed metrics to optimize it? Why why would it need to be delayed? Because you know usually the feedback is from the user, right? Like delayed. >> Yeah. Well, then >> yeah, that's right. You it basically be added to the data set and then you would use the latest optimize and just keep keep optimizing off of that >> ground truth data set. >> That's right. >> You will collect the outputs of your optimization and feed it back and the loop hits. >> Yeah. But that Why you're trying to do offline optimization, right? >> Yes. >> But I'm I'm asking, can you do this online where with the metric feedback? >> If you're good if you're a good enough engineer, you probably do it. But >> I'm not I'm not recommending replacing ML models with like optimized DSPI programs for particular use cases. Maybe like classification is a terrible example. I recognize that. But for other other are other in theory, yes, you know, you could do something like that. Yes. But for for particular LLM tasks, I'm sure we all have interesting ones. If you have something that is relatively well defined where you have known inputs and outputs, it might be a candidate for something worth optimizing. If nothing else, to transfer it to a smaller model to preserve the level of performance at a lower cost. That's really one of the biggest benefits I see. All right, last last question. I've heard that uh DSPI is can be kind of expensive because you're doing all these LM calls. >> Um so I was curious your experience with that and maybe relatedly like if you have any experience with like large context in your optimization data set ways of shrinking those. >> Yeah. So the question was do can BSI be expensive and then for large context kind of how have you seen that? How have you managed that? The expensive part is totally up to you. If you call a function a million times asynchronously, you're going to generate a lot of cost. I don't think DSPI necessarily maybe it makes it easier to call things, but it's not inherently expensive. It might, to your point, add more content to the prompt. Like, sure, the signature is a string, but the actual text that's sent to the model is much longer than that. That's totally true. I wouldn't say that it's a large cost driver. I mean it again it's ultimately it's more more of a programming paradigm. So you can write your compressed adapter if you want that like you know reduces the amount that's sent to the to uh to the model. Um in terms of large context I it's kind of the same answer I think in terms of if you're worried about that maybe you have some additional logic either in the program itself or in an adapter or part of the module that keeps track of that. Maybe you do some like context compression or something like that. There's some really good talks about that past few days. Obviously, I have a feeling that that will kind of go away at some point where either context windows get bigger or context management is abstracted away somehow. I don't really have an answer just that's more of an intuition. Um, but DSP again kind of gives you the tools, the primitives for you to do that should you choose. Um, and kind of track that state, track that management over time. So, I think that's it. We're going to get kicked out soon. So, thanks so much for your time. Really appreciate it. [music] [music]