Steve Koliski: At Stripe, we're landing about 1,300 PRs that have no human assistance besides review per week. A lot of where our work begins is it could be in a Google doc as we're planning a new feature or maybe a Jira ticket comes in or we're talking about something in Slack. I can click an emoji and then the menu will sort of attempt to oneshot resolving that prompt using all the tools that are available at Stripe.

Claravel: When you're in larger organizations, there's so much friction that can come between a good idea and getting it into the world. Not only can I have one of these, but I could have many, many of these running in parallel in isolated environments making isolated changes all at the same time.

Claravel: How are you getting all this code review done?

Steve Koliski: Whether the text has been written by Steve or the text has been written by Steve's robot, you still want that CI environment that's providing confidence that the code that's being changed is safe and that as it rolls out, we're having blue green deployment so you can roll back too. All of that is super critical independent of the nature of the authoring of it. No matter how juiced these laptops are, you get three or four work trees in and like it starts to sound like an airplane taking off, it's no good. And so I do think on this multi-threading agentic engineering work, cloud environments and virtual environments are so important to unlock velocity.

Claravel: Welcome back to How I AI. I'm Claravel. Today we have Steve Khaliski, a software engineer at Stripe, and he's going to show us how the Stripe team deploys a bunch of minions to do their engineering work.

Claravel: So, tell me what has been the effect that minions have had on you personally at Stripe and at the Stripe team as a whole?

Steve Koliski: Sure. So, you know, for me personally, um, I think sort of anecdotally, I don't remember the last time I I started work in the text editor, right? So, I do end up there often. Um, but, you know, what I found is that, you know, a lot of where our work begins is, you know, it could be in a in a Google doc as we're planning a new feature or maybe a Jira ticket comes in or we're talking about something in Slack and those are sort of like the more natural entry points to starting work, right? And then you end up in a text editor when it's time to, you know, actually do the work or make the final tweak. Um and it's just felt very natural. Um and it's think in particular the sort of like activation energy of starting work feels a lot lower, right? So if you know you're in a Slack thread and maybe there's a a piece of user feedback and it's something simple like a you know we have to update the docs or or maybe it's something more consequential and we just want to build a prototype. I can click an emoji and like the work begins and often the work finishes too. At Stripe we're landing about 1,300 PRs that are uh have no human assistance besides review per week. Um but at the minimum the activation energy of like starting to write code, seeing tests pass, maybe a test fails occurs without me even, you know, participating and then I can jump in and I can tweak and I can kind of like have that that momentum sort of uh sort of like generative momentum, you know, that I can I can hop in halfway through.

Claravel: What I think is magical about this is I love that concept of activation energy going lower because when you're in larger organizations, there's so much friction that can come between a good idea and getting it into the world.

Steve Koliski: When I as an engineer sort of in pre-AI time uh you know want to make a modification to Stripe um well Stripe is a huge codebase with tons of services um it it can't run on my computer alone so Stripe already has a long history of of you investing in great developer tooling having hosted development environments that I can spin up that you know have all the code already there and services running and I can SSH in and make modifications um and we have a ton of great CI tooling around that so that's the context. The idea with a minion is that I can provision one of those environments seated with a prompt and then the minion will sort of you know attempt to oneshot resolving that prompt using all the tools that are available at Stripe, right? So uh all of our internal documentation, our internal CI, our um you know test data, so on and so forth and it will loop through that in attempt to you know solve that prompt.

Claravel: One thing my friend Zach from LaunchDarkly said was look, what's good for the developer is good for the agent. So, there's this virtuous loop of if you have or do invest in developer experience for your human engineers, your agents will benefit off of that. And in turn, if you invest in developer experience or agent experience for your agent engineers, your development team benefits from that. And I think if you attach it to an AI initiative, that's like the secret way to get some of that good stuff.

Steve Koliski: Yeah. I mean imagine you're, you know, some code bases are small, but you know, Stripes is huge. Imagine you show up day one and there's there's no documentation and there's no tools and they say, "Good luck." Like anyone would have trouble and even if you threw the agent at it, it's very likely that the the context window would be blown by the whole codebase. Just scanning through it to understand all the intricacies would be like impossible or extremely expensive. So if there's a very blessed path for 90% of the common activities in being a engineer at Stripe that makes the propensity that the agent succeeds really high too. So good docs for developers are uh equally important for the agent.

Steve Koliski: So we've now transitioned from booting up the development environment to now we're in the the first agent run. So we have that prompt that I posted in Slack here. And now what it's going to do is boot up an instance of goose that's basically the harness that's you know going to run through all this.

Claravel: I love your system prompt. So sophisticated. It says implement this task completely. Colon and then just whatever you put in. No mistakes. I think people really think they have to overarchitect their initial prompt. And I think if you have a great harness, it can go a long way to extracting out a successful outcome from a pretty loose prompt.

Steve Koliski: I think now most things that could have an MCP server have an MCP server. So we're able to interact with a lot of the internal data we have. In the particular case of Minion um it's very specific to like the stripe developer experience and the stripe developer environment and we had been uh experimenting with goose early on and I think in this particular case we forked it to make some modifications as well and really what we were looking like sort of a base harness and loop um to apply all of our own tools and software to.

Steve Koliski: So, the demo we just showed is how we're thinking about using AI internally to accelerate our product development and engineering. The second way is thinking about how we're supporting all these businesses that are leveraging AI in their own products. But there's a third side which is like this sort of idea of agents as economic actors or agents that can spend money as part of their attempt to solve a prompt. We imagine a future where third party services are going to want to sell into these kinds of experiences and that those interactions will cost money. So we have to equip uh our agents with the capacity to spend so they can not only consume tokens, but so they can also pay services as part of achieving uh the prompt.

Steve Koliski: I told it to research Jen Lee, figure out what would be a good idea for her birthday, find a place, send invites, and then donate to Stripe Climate. The first thing we're going to do is that we've actually paid browser base to create a new browser session. It found out she's a matcha obsessed baker. Now we're interacting with this service called postal form. Postal form will take a PDF and actually send it in the mail. We actually made a $1.65 donation to Stripe Climate to erase 4.44 kilograms of carbon based off of our 70k token usage.

Claravel: What's really interesting about this particular example is it makes it very clear the economics of doing something agentically. It also just calls out like this actually does cost you in tokens whether or not your agent is doing outside transactions. So we're already operating in an economic framework, right?

Claravel: When AI is not listening, when your minion does not oneshot, what is your prompting strategy? Do you gentle parent your AI? Do you bribe it?

Steve Koliski: This sounds crazy but like I have made a concerted effort to always be polite. I definitely don't want to be caught being rude even though you never know. But the more serious answer is um one, asking it to explain or justify itself um has helped quite a bit and then I think in other cases where I know the right direction to go. I will start going in the right direction and then I will ask it to look at sort of like the git status to look at the diff or like look at other sort of like breadcrumbs that I've left as like the directional thing to help guide it.

Claravel: Got it. So, you're doing like the dad teaching his kid to ride a bike move where like your hands on the back of it and then you let it go.

Steve Koliski: It didn't really hit me until you said that, but there's something really weird about raising kids at the exact same time that the the robot emerges. That hadn't really clicked with me yet.