Nick: Today we're going to talk about AI at RAMP. And I'm going to give a quick introduction into what RAMP is. Um really briefly we're going to walk through the simplest possible expense use case that you guys can all resonate because I see everybody's drinking coffee. And um then we're going to talk quickly about a lesson that we learned this year while we were building gazillion agents um and sort of the pivot in the paradigm that's happening especially after February 6.

Nick: And um then we're going to double click onto how we built one of our most popular agents, the policy agent. Um and then finally we'll dig in into the infrastructure build that this is requiring to do on our side and in my mind most importantly the culture shift that needs to happen on everyone's teams in order to be able to operate in a way that delivers products into the hands of your customers in the fastest and most impactful way. Uh so without further ado uh quick intro about RAMP. Uh we are number one finance platform for modern businesses. We have 50,000 plus customers and we're in the business of saving you time and money.

Nick: So a cup of coffee usually takes about 15 minutes of your time because you got to do this three simple things which unfortunately take minutes. This compounds through the company and what RAM does in the simplest possible way we just condense time and return money back. Uh so a simple story of a transaction from tapping the card to writing a memo to classifying the transaction according to your GL to sourcing the receipt attaching the receipt um normalizing the merchant to your inventory of merchants is all done agentically at ramp.

Nick: We intentionally last year allowed each individual team to go and experiment and we ended up maybe with four different ways of doing the same thing. Um but instead you want to drive your framework towards a single agent with a thousand skills. Traditionally software would focus on only APIs and actions. In the new paradigm software is doing everything. So you want to focus on building an autonomous system of action that can react, reason and act without a human or with very little human supervision.

Viral: So, a lot of finance teams are looking at receipts like this basically every day and maybe they might have hundreds or thousands of these. If you told me to look at this and decide if I should approve or reject this transaction, I'm probably going to make a mistake. So, policy agent basically reasons on this image and all the transaction data that we have and told me that there were eight guests in the receipt. It was below the $80 a person cap that we have internally and so policy agent told me to approve this transaction.

Viral: We kind of saw this as an opportunity to actually take out a page from Andre Karpathy saying that English is the new programming language and kind of turn the expense policy into the rules themselves. One of the main things that we realized across ramp is that we really needed to lean into the fact that AI products cannot be oneshotted. You need to start with something simple. One of the early learnings was that a lot of the reason that policy agent would be wrong would be less on the models themselves and more about the context that we were giving to LLMs themselves.

Will: When we first started building the policy agent internally, we went big. Let's automate all of finance. But when it came down to it, we actually had to start small. The simpler the system, I think the easier it is to iterate on top of it. As you go from simple to complex systems, your autonomy goes up, your AI seems smarter. But in exchange, you're losing traceability and explainability. A smaller black box becomes a bigger black box as the system becomes more complex.

Will: So one thing that is really important from the beginning you need really good auditability. Turns out the users are actually incorrect. They're wrong. So, we had to define our own definition of correctness. Um and to do that we had a weekly labeling session with across functions. We decided let's just build our own [labeling tool]. We used Streamlit. Non-engineers can go and personalize it.

Will: With the ground truth data set, we were able to make quick iterations. You had evals and it's pretty important to have them early on. If you run it as part of your CI, then everyone now can safely merge in code. Having evals really helps make confident model changes. Online evals are also great. For us, part of that was how many unsure decisions we had, which just meant that the agent didn't have enough information.

Ian: The core of how most of applied AI happens at RAMP is our applied AI service. There's really three main extensions: the first is structured output and consistent API and SDKs across different model providers. The other thing is batch processing and workflow handling. And then the last which is a pretty big deal is the ability to trace different costs across teams.

Ian: We're pretty big believers that it all comes down to the catalog of tools that teams are building. We're up to many hundreds of these tools today and Nick thinks that this could be multiple thousands over time. We decided to build our own internal background coding agent, which we've called ramp inspect. Currently this month, Ramp Inspect is responsible for over 50% of PRs that we merge to production.

Ian: Let's pretend we have two different teams. Team A care about impact, handle ambiguous problems, adopt new tools and obsess over the user experience. Team B debate libraries, bike shed the details, and focus on performative code quality. There's going to be divergence.

Ian: Coding was never really the hardest part of a lot of jobs. You're really compensating [senior engineers] for the judgment that they bring to the table. You could still build the wrong thing just a lot faster. Companies are just going to chase opportunities they couldn't afford to pursue. You're going to rebuild systems that are too expensive to touch. Software is perpetually not finished.