构建跨网络运行的金融浏览器智能体：Browserbase 与 Stagehand 的架构与实践

Show of hands, who knows these three companies? Ramp, Bill, Stripe. They familiar with the crowd? Cool. When you think about what these three companies have in common, well first they're all financial institutions, they're dealing with highly sensitive information, but when you think about it for like Ramp, Bill, and Stripe, a lot of this information lives on the hardest to reach parts of the web, whether that is behind a portal where you need to log in, whether it's on a website that is heavily protected by antibot. It is extremely challenging to get access to this information, and the answer is using browser agents. So, if we want to start off, we'll go with Ramp first.

So, Ramp, I'm sure everyone in this room is familiar with Ramp. We actually are big users and fans of Ramp. Uh they make it incredibly easy to for a company to manage spend. When they were growing and scaling, they started having to interact with hardest to reach portals, and more portals than than ever before. And so, the problem or the gap was, how can we fetch these receipts efficiently at scale?

They didn't know about the browser agent. So, instead, they're like, let's just put together a team of humans, and they will log onto these portals, and they'll go fetch these receipts. Well, that can that can be good for a series A, series B company, but when you start growing and scaling to the to the masses that uh Ramp is at, you need a more efficient way. You can't continue to hire humans. You're just going to be bottlenecked by the limitation of scale.

So, in comes browser agents. And now, again, we are big users of Ramp. I booked my flight to this conference using my Ramp card. Immediately after booking my flight on that Ramp card, Ramp spins up a browser agent. That browser agent goes to Delta. It logs in. It captures that receipt, and it matches it to the transaction. Um so, once they do that, that's that's all good and dandy, but when you're doing it at scale, you're going to run into things like antibot. So, how do you reduce that friction? How do you eliminate getting blocked?

And that's when we scale beyond the laptop. We need to replicate it. For companies like Ramp, they are not going to have thousands of laptops open running a different human version of Chromium. They're not going to have one person with a cron job running this either. They need scaled infra that lets them get to sites reliably. The problems start to appear though when you put that same Chromium in the data center.

Um let's say we do this. Let's say we've put that in. The problem now is when we launch a called out browser, that headless browser in an an an EC2 instance somewhere. Every single time we launch it, it's going to have a different fingerprint. It's going to have fresh cookies. You may need to log in again. And proxies. It's going to clear indicator that you are coming from a data center instead of a residential IP. And sure, you could trick the website by using a proxy service and changing your proxy. And that means is you might not be presented as many captures or you can fake your location from that data center, but that still doesn't change the underlying problems with Chromium. You're using that same Chromium on a laptop.

And if we just step one step down and go a little bit deeper and think about what makes Chromium so good is if you look at the browser process, right? Inside you have a page engine. This is a very simplified version. And where it renders the content and it has all the HTML, JavaScript. Um then you have your network and if you ever like inspected before you go on a network tab and you can see all the requests coming in and out. And you have your device and graphics. The engine is so great that they can make a change every 2 weeks, which makes it terrible for us as a company trying to automate the web.

So we forked it. And I could go on forever on how many changes we made to the Chromium binary. Um but we forked it and came up with a verified browser that we run in the data center. And what we did with three things I'm going to share now is we first removed the automation flag. An obvious one, right? When you're driving a headless browser um through an automation, you will pass this webdriver equals to true and we just swap that to false everywhere. Then we aligned on the identity. Instead of now being called headless Chrome, we're calling it Chrome. Now these may seem like simple fixes, but because how complicated Chromium is, this goes deep into the code and it's in so many different places that we need to find. And every 2 weeks is a new change. That's why we have our own version. We also [clears throat] keep the process alive. What I mean by that is a lot of scraping jobs or automation jobs or workflows that people are doing as developers locally or even in the hosted infra will start and stop. They kill the browser and they kill a Chromium instantly as soon as the task is done. Instead, we leave chromium running and close the windows out.

So, why don't we put it to the test? We've got a snippet here and I'm going to ask Aric, can we launch a browser? Okay. And bear in mind the Wi-Fi has been terrible today. So, it's launch is going to take a second to render and we are now looking at an iframe of a live view of a hosted browser on BrowserStack. That's great. We can go to Rome. Okay, nothing special. Let's stop that. And let's go to the next slide. Why don't we launch 100 browsers? Do the exact same thing but we'll launch them concurrently. Okay. And to the next slide. Um now what's going to happen in the background is I'm calling the same thing with a bunch of different websites and you will be able to Aric scroll through the um some of the sites and we're not automating at the moment. We're just going and visiting the sites as 100 browsers.

Now I'm going to try something for the next minute. Um I set this up very quickly and the Wi-Fi is a little sucks a little bit. And it's all through an Engrok portal on my laptop. So, if it doesn't work, uh we'll move on. Um I want everyone to scan this QR code and you will be able given a random website out of one of these 100. Um someone tell me if they can see a browser. I'm trying to prove to you that these browsers in our cloud and you have access. Someone shout out when they're when it's loaded.

When we talk about Bill, so Ramp scale and access. Now, it's about control. Everyone's familiar with Bill. Um Bill, you know, processes a ton of payments across thousands of different vendors. When you think about using agents to do that, the biggest challenge is well, every portal Next slide, please. Every portal is different. The checkout flow is different. The process to fill out the payment is different. The UI changes. So, in reality, the way Bill thinks about this is it's impossible to automate with agents. That's why we need a team of humans to go click through portals and do this on behalf of our customers. Um and again, the answer to this is an agent that can use a browser. Um and so, for Bill, the notion was how can we build a single agent that can interact and act upon thousands of different portals. One agent, thousands of different portals.

I want to start with understanding again what is wrong with Playwright. Playwright's great. I first learned how to code by using Selenium and beautiful soup and writing Playwright scripts by hand. This was before AI. Now, I can't remember last time I coded. And so, with Playwright, what the problem is is as soon as that website changes, the button moves, the text box changes like from first name to just name. In the in the back, like when you're looking for that XPath in Playwright or you're trying to click on an element or a selector, your script's also going to break. And now, that's fine if you're doing one-off tasks and you're running everything locally, but when you try to scale like a company like Ramp or Bill, you need reliability every single time. The reason we like built Stage Hand was because we wanted to give the ability to agents to navigate the web and control them. Next slide.

So, how does it actually know? What's different? Well, if you play play Sorry. What's different? So, Stage Hand actually works a little bit like Playwright in some sense. You can use deterministic rules and go page.click or everything you'd see in a Playwright script. But, when it gets a little bit more complex, you can actually send natural language prompts to go and do actions on the site. Pay page.click or page.act, go fill out this form, or enter my name, or enter password, find the red button. Stage Hand will then be able to take the DOM and accessibility tree and everything it sees on the website, pass it to an LM, pass it to an agent, and understand where to click and go and click it.

Now, you may be thinking it might get expensive, but that's where we have our excellent engineers behind Stage Hand. And to demonstrate this quickly, Eric, if you could hit that compare button. If you were to use an agent with playwright, you'd be sending everything on the left. Uh, that's 122k tokens. And that's a raw HTML. And you ask your agent, "Where do I click? Where do I go to?" What Stage Hand does before it hits that agent is it shrinks the DOM to a really cool accessibility tree, where we're only sending like 4,000 tokens for this same example. And it'll give direct selectors. So, Stage Hand will go, "Click that." Save a lot of money on um the amount of tokens that you're sending, but also reduce the context and get quicker results.

And if we go next, another reason why we use um Stage Hand and with Browser Base is because there's fewer net There's fewer round trips. Let's say we are currently controlling a browser in a data center. We've got to talk back. We send the HTML back and forth every single time. What we do at Browser Base is we put Stage Hand next to the browser. And that What that means is Stage Hand is a Chrome extension on a hosted browser. So, those actions become a whole lot quicker, and there's no round trip latency from those actions and those like instructions from the agent of where to click and where to go next. And so, when you bring these two things together, you need scale from the browsers, Stage Hand gives you the control, and we're going to show you an example that use both.

Next slide. Now, I'm going to show a couple more demos. We're going to start with one company website and we're going to go to Chipotle and find out what it sells using Stage Hand. Um and next slide then. Okay. So, this browser will now spin up on a hosted browser and browse page and it's going to use Stage Hand um extract to extract it. And there we go. We've got Chipotle Mexican Grill. Um it's selling some lime chips. Cool. Uh if we press stop. Now, this is where it's going to get fun. Let's go for the next one. And now we're going to research four different sources for Chipotle. We're going to go to their website, investor page, SEC filing, and New York Stock Exchange listing.

Okay? So, we're running and all right, you can also scroll once they load. Okay, great. So, we've got the SEC filing down at the bottom, and we've got New York Stock Exchange, and up top I think we've got the website that will appear in a second. But, there we go, they got investor relations, as well. So, these are actually happening quicker than they are on the screen. This is already being happened in in the back. It's just the live view that's streaming back to us. And obviously, when you're running so many browsers at scale, you're not always going to be checking the live view. This is more for development.

Okay, cool. Let's stop that. And let's go to the next one. Now, we're going to do 20 companies for the same three sources. And we're going to hit run. And bear in mind, this is the same code that's running on every single website. Then, we go to the next slide. And we can see Chipotle will come up at the top. I'm getting hungry now. Um JP Morgan. Um there we go. All right, keep on scrolling. Let's see what else is there. Golden Sachs. Going and doing the same tasks across every single site without interruption, unblocked, all using Stage Hand as well. And it's really cool cuz it's a verified browser that I mentioned at the very start, which means that you don't get hit with as many CAPTCHAs. And if you do, we solve them. And the Stage Hand allows us to navigate without anyone looking at this.

Okay? And we can stop there. And I think we can see some results on the right-hand side. Um next slide. Okay, cool. We've got some of the results here. Um SEC filing data. Keep on scrolling. I honestly have no idea what the SEC stuff means. I'm not a finance guy. But, hopefully it's useful. Cool. Um let's go to the next slide. And so, that SEC template I was something I built a while ago. If you want access to that particular, just the SEC one, you can do. Um, but we also have if you want to get started with anything browser-based or anything I showed, we have if you go to our website, we have a copy prompt which you just send it to your agent and that'll do a lot of the work as well. Uh, next slide. Okay. Thank you so much. That's our talk.