Um but I want to talk a little bit about the first wave of AI and it's it's been a fun one. I think the first big wave and I'm sure we've all experienced this is just how we build software. Um but there's some sort of net effects of that. It's a lot of bigger PRs that are coming through. We definitely see a lot of this a lot more frequently. So people are shipping code at a much, you know, faster rate from developers and we're beginning to see maybe from even non-developers that maybe don't actually know the code or what it's doing or sort of like operating principles behind it. But we're getting developer productivity. And that's good, right? That's that's a good thing that we're all able to sort of produce more and faster.

Kind of sort of what we actually found out and this was a survey study done is that 70% of the time from an engineer is actually not just like is not focused just on writing code. It's actually spent on actually running the code that is actually shipped into production. Um maintaining all the the platforms, scaling the infrastructure, debugging all the incidents, and being on call, um shipping hot fixes, right? Dealing with alerts, um updating all the sort of run books and operating procedures, restoring services, dealing with escalations, dealing with sort of like um questions from other, you know, teams and things like that. So, really coding was never the the the big bottleneck, right? Um a lot of it was really around, uh thank you, granola. Um a lot of it was really around like how do we check how do we actually run these things sort of in production. And that's getting harder and harder and harder. Um AI is creating a lot more issues in production as, you know, AI code sort of goes through. Um it's not clear we have the right sort of um structures in place to deal with the amount of kind of changes that are coming through. Um unlimited tokens is is sort of coming to an end, the the token max, right? They're starting to clamp down. Prices are going up. Companies are getting a lot more stringent on, you know, what's being used um for AI. Um you know, we need full stack AI. It's not just about the models anymore, it's about the context around the models and what the models can do inside of a specific domain. These become the the problem areas that we need to sort of uh focus in and tackle on. And this is true today, right? So, it's it's creating more sort of uh complexity inside of our environment. But, I mean the the reality is that systems have always been complex. That's why we have, you know, these big tools that can, you know, try to give us insights into these systems. Um there are multiple teams, there's multiple systems that are all having to work together, and they all have their own, you know, goals that they're trying to deliver towards, but you have organizational goals. And how do you keep all of this sort of uh you know, um in balance, right? Um how do you, you know, pull all of this stuff together in a way that actually uh actually helps you and and facilitates your uh your organization. Um and the answer is, well, you got to use AI inside of production to deal with um sort of the the amount of increase of complexity that AI is kind of putting into your product or into your system. Um and so that's where Resolve, you know, this was kind of our sort of hypothesis from the beginning was um you know, we're going to see an influx in um you know, issues uh coming out of coding um just the increase in coding uh velocity. Um there's going to be more need for kind of AI to actually operate and run these run these systems. Um we're, you know, lucky to work with, you know, some uh world-class engineering teams that are solving like really difficult problems at, you know, crazy scale. Um and you know, that that gives us insight into a lot of how bigger organizations are having to deal with the influx of AI, etc.

And then we also focus a lot on background agents. And this is sort of covering the long tail of, you know, what happens when there's not a fire brewing um or or going on at any one, you know, point. Um there's still lots of operational work that you as an engineer or an engineering team have to do and a lot of ceremonies of, you know, passing contact context off or dealing with kind of one-off issues or kind of having to scratch that itch in the back of your head of like, is that part of the system okay or not okay? Um and you're constantly having to sort of balance across all the these different things. Um underneath all of that, you know, the we have an agent uh architecture um deals with models and context and reasoning and and actions. Um learning is a a I'll sort of like half pause on that one. I think, you know, some of the biggest issues that we've seen, it's it's not that a a model by itself is is not smart or whatever. I mean, models have gotten incredi- incredibly capable over the last um year, let's say, right? But especially over the last like 6 months or so. Um but the idea of understanding, like truly understanding your environment um and the way that your services interact and where the hotspots are, keeping track of all of that sort of understanding is incredibly difficult. But it's incredibly important for any model to be successful at, you know, the task that it needs to do. It has to have an underlying sort of learning system to be able to capture that knowledge um and that sort of understanding of how your system operates. Um so we spend a lot of time thinking about how do we have systems that not just can understand your environment at any one point, but grow as as your system evolves? Because again, your system is evolving faster and faster. We need to keep up with learning about what's the current state, um what's the current sort of causal chains that we need to be sort of keeping keeping an eye on. Um and then of course all the enterprise enterprise stuff sort of underneath. Um and so this is that same view kind of uh packed out. Um Come on. Okay. Um, but so today so we do a lot at Resolve um, the on-call and the incident stuff. I'm going to focus a lot uh, more just on the background agent um, stuff. So how do we deal with the things that maybe aren't sort of immediate fires. Um, if you have questions about the immediate fire stuff, um, we have a booth down in the expo. Please come check it out. Uh, our team would love to demo to you etc. Uh, but today we're going to focus on the background agent.

So kind of a little pop quiz. Um, feel free to raise your hands. Is anybody using agents in part as part of your daily workflow? Maybe outside of the coding. I'm assuming everybody's doing coding agents these days. Is anybody doing like actually running sort of um, agents that are sort of helping in other ways? Okay. Decent amount. Any good examples? Any fun stuff that anybody has? You can just yell it out.

Meeting review. Meeting reviews? Yeah, meeting reviews. I just had my granola show up and Market research. Market research. I do a lot of. [laughter] So let's let's have a good conversation about. Yeah, yeah, yeah, I do that all the time. Any other ones? Maybe one more? Therapy. What's the one? Therapy. I can't hear it. Therapy. Therapy. That's a fantastic one actually. We're we we are humans here today. This is very important. Um, that's actually that's a very good one. Um, okay. So so people are having some stuff uh, going on. Um, So, you know, and this is kind of recaps a little bit uh, again. A lot of production work is not about is not there's not a sort of big ceremony that everyone is focused on for the the type of work that we have to do. Um, on-call you you've got a page that goes off. You know somebody's going to receive that. Incidents you create a bridge, you invite people in. That's great. Um, but there's just a long tail of other things that we are accountable for that doesn't have sort of a thing that's going to show up in your sort of job description of like this is what you're going to be, you know, responsible for. Um, watching deploys that go out and make sure that they're actually getting out um, healthy. Um, a morning report uh, incident digest of just like what's the state of my system today so that we're all on the same page. Um, hey that P99 drift kind of came back. Is somebody looking at that or not? And you know, this is pulling people in to to try to like figure out what's going on. This may not be paging, right? Because we don't we're not going to alert on everything. Produce the capacity report, right? Like are we tracking okay, right? This is maybe a company goal this this quarter. Are we tracking against that? Somebody's going to have to be responsible for doing that. The recurring health check and just kind of checking and making sure things are kind of running okay and not waiting for a customer to come complain first. So this work doesn't have like an obvious like, oh, this you know, this now needs to go be done. But it's work that we end up having to do.

So what is a task? A task is just execution and the context to understand how to actually execute the task. Execution is very very important. It's understanding what to do and being able to execute that. Maybe having access to the tools, etc., right? Obviously very important to do. But we think the production context is just way more important because it's one thing to go check a dashboard. It's another thing to say that metric smells off. And the execution is can load the dashboard. It's the production context that's going to say, this feels wrong. And I don't know if I can even explain why it feels wrong. It just feels wrong and I want to dig into the next layer of sort of understanding of that. And so really if if we start talking about background agents and being able to perform task, you need both of these. You need the execution engine, that's great, but you really need that production context that tells you is this important or not important. So every background agent, you know, there's a a few different principles that we like to think about with our background agents. When does it work? How does it work? How does it know what to go do? When does the agent work? It can work in a bunch of different ways. It can just do it on a schedule. Maybe this is the the report, etc. Just kind of do some summarization for me kind of on an ongoing basis. Maybe it's a weekly event, right? We do an on-call handover uh every Thursday, and so that a lot of um the work that our agent does is sort of prepare like what what are the kind of interesting trends from the last week that the next on-caller needs to sort of understand as they pick up the rotation. Um, event streams, so you know, there's lots of systems that will sort of push events as kind of key things happen. Um, so deployments go through a CI ICD pipeline. Um, there's other sort of uh Slack-based, right? We get a lot of uh Slack things messages coming through, um etc. Um, and these are things that we can sort of pick up and trigger and say, "Oh, if this event happens, um let me sort of understand what that event is and go do some work." Um, and then message-based, so I can just tell it, "Hey, go do some work." And it will go do some work. That's fantastic. Um, how does it run? Always runs. It's in the cloud. Um, so if you close your laptop, it's okay. Um, runs inside of a sandbox, so it has kind of a file system underneath it. Um, this allows it to sort of self-organize a lot of its work, etc. as it's doing uh doing things. Um, and then obviously back to the learning loop, right? So, that idea of knowledge and sort of a memory system underneath that um to really understand your systems and as it's doing a task, able to sort of reflect on that task and uh you know, do a better job next time. Or the things that it learned from one task, it can sort of apply into a different task. Um, because again, this this sort of shared uh sort of knowledge system um works across all the different tasks that we have. Um, so how does the agent know what to do? Um, it has a task system. It can pull in all the skills that you have in other systems, that's fine. You can connect those. Um, and it's got obviously the integrations that it's going to plug into.

So, deployment monitoring. So, this is a big one, you know, any change inside of your environment is an opportunity for something to go wrong. Um and so, you know, having an agent that's able to watch as all these change events come in just to do a sanity check of is everything stable is incredibly incredibly important. And, you know, a lot of people have decent CICD system. I mean, this is like tried and true stuff that we've had as an industry for quite a while. But, we we noticed a couple gaps, you know, from in most of our customers. You know, typically the checks that it does are good. They're good baselines, but it's not exhaustive based on the type of changes that are going in etc. There's certain signals you'd want to watch or not want to watch. And so, every rollout is a bit unique. Often times you have change systems that you're not piping through a CICD system like a feature flag or maybe some infra changes that might happen which maybe don't get any monitoring at all. And you're sort of just trusting that an alert might fire and an on-caller will wake up and say, "Who changed what?" right? Deployment monitoring is is actually a really big use case that we suggest people sort of go through and I'll show some examples of that in a second.

Schedule health and anomaly checks. So, this is just sort of the ongoing periodic checking of some of your systems. And, you know, this is maybe something where it's like go check, you know, sort of my general dashboards on a routine basis maybe every morning just kind of do a casual check just to make sure there's nothing kind of weird from last night that I might need to be be of. But, uh, this can also just be sort of a time-based thing. Like I made a change in part of our system. I'm worried about this, you know, uh, you know, third-party service that I'm kind of interacting with. Let me just kind of set an agent to kind of watch that maybe for the next week just to make sure everything is kind of stable and then that agent can sort of, um, you know, stop his job.

Uh, operational reports and handoffs, I talked a little bit about this. Uh, these are the sort of ceremonial things that we might want to do just to, you know, spread information, summarize things, kind of bring things to the fore.

Um, and then a first responder to engineering questions. And this one's kind of fun a little bit, um, because the trigger for this is actually just a Slack message and I will say, um, one of my biggest, like, let's call it responsibilities, uh, as an engineer is watching all my Slack channels and trying to make sure everyone's kind of happy. Um, and like that nobody has any burning questions or anything like that. Um, and so I can be sort of heads down trying to build something, um, and then, you know, the, you know, uh, eventual sort of Slack notification comes in that like this channel somebody asked this sort of kind of important question, um, and I just need to jump in there and and try to provide context, etc. It's not hard work. It's not hard for me to go answer questions, but it's disrupting me and if I don't go answer it, um, they won't get an answer for a while. And what we found is like our agent actually has access to a lot of information that people ask questions about. At least this is true internally. Um, so we actually have an agent that can watch all of these sort of critical channels, um, and determine whether it has enough sort of confidence to answer the question or not. Um, and one of the fun things is like it, uh, the our agents have access to like Slack DMs and things like that. Um, and so you can have an agent that basically will DM you to say, "I think I know the answer to this, but I'm not sure. Can you confirm this for me before I, you know, respond back?" Um, so this kind of emergent behaviors gets kind of fun and interesting as you just kind of build these things out.

Okay, so I'm going to flip over and hope all of this works. Um, Cool. Um So, uh let's see if I can find the one that I wanted to show. So, this is our uh sort of demo application running in our sort of demo sort of Slack environment. Um and uh what I wanted to show off was some of our deployment stuff and talk a little bit more about um what's kind of going on under the hood. Um so, this is, you know, sort of fake environment um just to kind of showcase some things. Um so, here um anytime somebody posts a sort of GitHub tag, ... [truncated for brevity in translation logs] ... but thanks for coming. Appreciate it.